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Executive  Summary 


This  report  is  written  from  the  perspective  of  the  National  Institute  of 
Standards  and  Technology  (NIST).  It  is  a deliverable  against  the  1992 
Computer-aided  Acquisition  and  Logistic  Support  (CALS)  Statement  of 
Work  NIST  has  with  the  CALS  Evaluation  and  Integration  Office  (CEIO).i  It 
contains  both  tutorial  information  and  issues  associated  with  the  various 
types  of  testing  activities  under  the  purview  of  the  CALS  initiative. 
Others  may  benefit  from  this  report  since  it  additionally  provides  general 
tutorial  information  about  the  various  types  of  testing  activities  and 
associated  terminology,  and  summarizes  conformance  testing  activities 
for  several  national  and  international  standards. 

The  four  primary  testing  activities  discussed  in  this  report  are:  standards 
testing,  component  testing,  conformance  testing,  and  acceptance  testing. 

Standards  testing  determines  whether  the  national,  international,  or 
military  standards  (and  specifications)  are  viable  and  implementable. 

Component  testing  is  conducted  to  verify  the  implementation  of  the  design 
for  one  software  element  (e.g.,  unit,  module)  or  a collection  of  software 
elements. 

Conformance  testing  tests  the  extent  to  which  an  implementation  under 
test  is  a conforming  implementation. 

Acceptance  testing  determines  whether  a software  system  satisfies  its 
acceptance  criteria  and  enables  the  user  to  determine  whether  to  accept 
the  system.  This  includes  the  planning  and  execution  of  several  kinds  of 
tests  (e.g.,  functional,  interoperability,  performance  tests)  to 
demonstrate  the  implemented  software  satisfies  the  user  requirements. 

Where  two  or  more  people  are  gathered  to  discuss  the  topic  of  "testing," 
there  are  usually  as  many  definitions  of  testing  as  there  are  people  in  the 


1 The  opinions  and  recommendations  of  this  report  are  not 
necessarily  those  of  the  CALS  Executive  or  the  CALS  Evaluation  and 
Integration  Office  Management. 


room.  To  emphasize  this  bedlam,  a few  key  testing  terms  and  their  many 
definitions  are  highlighted:  certification,  validation,  verification. 
Because  terminology  is  important  for  communicating  in  the  testing 
community,  an  extensive  glossary  of  terms  has  been  added  as  an  appendix. 

Key  to  this  report  for  CALS  is  a section  devoted  to  several  of  the  testing 
issues  facing  the  CALS  initiative; 

Whether  the  CEIO  should  recognize  any  acceptance  t^s+'ng 
performed  by  the  systems  integrator  prior  to  government 
procurement. 

Whether  it  is  cost-effective  for  the  CEIO  to  fund  a validation 
testing  system  for  the  Standard  for  the  Exchange  of  Product  model 
data  (STEP),  and  if  so,  where  should  such  a system  be  hosted. 

Although  NIST  develops  conformance  testing  services  for  some 
standards  which  are  Federal  Information  Processing  Standards 
Publications  (FIPS  PUBS),  no  means  exist  to  perform  conformance 
testing  on  the  additional  requirements  imposed  on  the  supplier 
through  military  specifications.  Should  NIST  be  responsible  for 
CALS  military  specification  conformance  testing? 

Accreditation  of  testing  laboratories  is  often  viewed  as  an 
expensive  overhead,  both  in  actual  dollars  required  to  initiate,  and 
time  required  to  establish  such  a program.  Should  the  CEIO  invest  in 
accreditation? 

- The  CALS  Test  Network  (CTN)  affirms  CALS  compliance  only  for 
DoD-owned  systems.  Is  this  appropriate? 

The  CTN  performs  standards  testing  and  interoperability  testing 
without  routinely  performing  conformance  testing.  This  CTN 
practice  is  not  the  same  sequence  that  occurs  in  general  practice. 

Should  the  CEIO  try  to  leverage  industry's  test  suite  and  tool 
development  resources  to  benefit  conformance  testing  and  reduce 
industries'  investment  costs? 


- Should  the  CEIO  fund  the  development  of  a conformance  testing 
service  for  the  FIPS  and  associated  military  specification? 

Even  if  conformance  testing  services  existed  for  all  the  CALS 
military  specifications,  how  does  the  CEIO  guarantee  enterprise 
users  will  take  advantage  of  such  services? 

For  each  issue,  alternatives  are  provided,  pros  and  cons  considered,  and 
recommendations  drawn. 
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I.  INTRODUCTION. 


A . Content  of  this  Report. 

If  the  CALS  initiative  is  to  be  successful,  government  and  industry  have  to 
attain  a high  level  of  confidence  that  the  CALS  specifications  and 
standards  are  adequate  to  satisfy  the  requirements  for  digital  delivery  of 
technical  data  supporting  user  applications.  Equally  ac  important,  both 
government  and  industry  must  be  satisfied  industry  implementations  can 
deliver  this  technical  data  according  to  a set  of  specifications  and 
standards.  For  this  to  happen,  CALS  has  to  influence  testing  in  three 
areas:  standards,  conformance,  and  acceptance  testing  [JCM091]. 

The  content  of  this  report  covers  those  testing  activities  which  have  been 
or  are  funded  by  CALS,  by  DoD  services,  or  by  industry  to  support  CALS. 
Although  specific  activity  descriptions  may  be  limited  to  the  CALS 
community,  the  types  of  testing  activities  generically  described  in  the 
following  pages,  the  author  believes,  satisfy  most  enterprise  users’ 
application  requirements. 

B.  Scope  of  this  Report. 

The  audience  of  this  report  includes:  the  CALS  Executive  and  supporting 
CALS  Evaluation  and  Integration  Office  (CEIO)  managers;  CALS/CE  Industry 
Steering  Group  (ISG)  participants,  particularly  those  who  have  invested  in 
testing  activities  relative  to  CALS;  suppliers  of  CALS  implementations; 
and  enterprise  users  responsible  for  implementing  CALS  solutions  into 
their  logistic  support  life  cycle  processes. 

C.  Structure  of  this  Report. 

The  body  of  the  document  is  divided  into  nine  sections  with  additional 
supportive  appendices.  The  following  is  a brief  description  of  each 
section: 

I Introduction.  This  section  introduces  the  need  for  testing  realized 
by  the  CALS  Evaluation  and  Integration  Office. 
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11  Terminology.  This  section  attempts  to  raise  the  reader's 
consciousness  on  how  nebulous  testing  terminology  really  is. 

IN  Background.  Background  is  divided  into  two  primary  parts:  an 

introduction  to  the  standards-making  activities,  then  the  types  of 
testing  as  they  relate  to  the  standards.  A listing  of  publications 
relative  to  the  types  of  testing  is  also  presented. 

IV  Testing  Activities.  There  is  a general  introduction  of  the  primary 

testing  participants  as  they  relate  to  those  testing  activities 

associated  with  CALS.  This  is  followed  by  brief  descriptions  of 
participants'  activities  relative  to  each  type  of  testing. 

V Issues.  Through  the  course  of  discussion  under  section  III  and  IV, 

several  issues  are  highlighted  in  bold.  Under  the  Issues  section, 
these  issues  are  examined  in  more  detail,  some  alternatives 

proposed,  and  the  pros  and  cons  associated  with  each  alternative 
listed.  Additional  issues  not  found  elsewhere  in  the  text  are  also 
included  here  for  discussion. 

VI  Conformance  Testing  Status  of  CALS  Standards.  Although  this 
section  is  a snapshot  in  time,  it  gives  a status  update  of  various 
conformance  testing  activities  for  current  CALS  standards,  as  well 
as  those  potentially  on  the  horizon.  For  some  standards,  additional 
system,  process,  or  programmatic  detail  is  provided. 

VII  Conclusion.  This  section  is  a recapitulation  of  the  report  as  a whole. 

VIII  Bibliography.  These  works  are  used  throughout  the  document  and 
have  provided  the  author  of  this  report  with  a source  of  expertise 
relative  to  particular  standards,  programs,  or  testing  activities. 

IX  Acknowledgements.  Beyond  the  many  references  reflected  in 

Section  VIII,  there  were  also  several  individuals  who  helped 

compose  graphical  representation  and  technical  interpretation  of 

their  particular  conformance  testing  service.  These  individuals  and 
others  who  have  provided  additional  assistance  are  acknowledged  in 
this  section. 
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II.  TERMINOLOGY. 


Where  two  or  more  people  are  gathered  to  discuss  the  topic  of  "testing," 
there  are  usually  as  many  definitions  of  testing  as  there  are  people  in  the 
room.  Add  personal  expectation  to  the  formula,  and  one  is  left  with  a 
nebulous  understanding  of  what  "testing"  provides.  To  emphasize  this 
bedlam,  a few  key  testing  terms  are  highlighted  below. 

Certification. 

There  is  a discrepancy  in  the  use  of  the  word  "certification"  between  DOD- 
STD-2168: 

"a  process,  which  may  be  incremental,  by  which  a contractor 
provides  objective  evidence  to  the  contracting  agency  that  an  item 
satisfies  its  specified  requirements"  [2168] 

and  ISO/IEC  (International  Organization  for  Standardization  / 
International  Electrotechnical  Commission)  Guide  2: 

"procedure  by  which  a third  party  gives  written  assurance  that  a 
product,  process  or  service  conforms  to  specified  requirements"  [2- 
91]. 

DoD's  definition  primarily  applies  to  the  quality  of  software  to  be 
purchased  and  meeting  the  enterprise  user's  specific  requirements,  i.e., 
acceptance  testing.  The  ISO/IEC  Guide  relates  to  the  formal  process  for 
judging  conformity  of  a product,  process,  or  service,  primarily  used  in 
reference  to  conformance  testing.  The  ISO/IEC  definition  is  reinforced  by 
a similar  conformance  testing  application  in  American  National  Standards 
Institute  (ANSI)  Z34.1-1987's  definition: 

"the  procedure  by  which  written  assurance  is  given  that  a product  or 
service  conforms  to  a standard  or  specification"  [Z34.1]. 
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Validation. 


There  are  several  uses  of  the  term  "validation”  both  within  the  CALS 
community  and  outside.  The  type  of  validation  testing  the  National  PDES2 
Testbed  (NPT)  and  the  CALS  Test  Network  (CTN)  performs  during  standards 
testing  is  closely  aligned  with  Webster's  definition: 

"an  act,  process,  or  instance  of  making  valid.  (Valid  - having  legal 
efficacy  or  force)"  [WEB79]. 

DoD-STD-2167A  and  FIPS  PUB  132  use  the  term  as  it  applies  to  software 
development,  the  component  testing  phase: 

DOD-STD-2167A  - 

"the  process  of  evaluating  software  to  determine  compliance  with 
specified  requirements"  [2167A] 

and  FIPS  PUB  132,  which  adopts  ANSI/IEEE  Std  1012  - 

"the  process  of  evaluating  software  at  the  end  of  the  software 

development  process  to  ensure  compliance  with  software 

requirements"  [FIPS132]. 

In  conformance  testing,  validation  is  performed  by  a third-party  testing 
laboratory,  described  at  length  in  the  American  National  Standard  (ANS) 
Z34. 2-1987  [Z34.2].  This  ANS  definition  does  not  include  issuing  a 

certificate  at  the  end  of  the  process.  The  NIST  draft  proposed  FIPS  for 
Conformance  Testing  Policy  and  Procedures  does  include  issuing  a 
certificate: 

"the  process  of  checking  the  conformity  of  an  implementation  of  a 
standard  to  its  standard  specification  through  conformance  testing, 
and  when  compliance  is  demonstrated,  issuing  a validation 

certificate"  [FIPS88]. 


2 Product  Data  Exchange  using  STEP  (Standard  for  the  Exchange  of 
Product  model  data). 
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A last  definition  on  the  word  validation  also  supports  a conformance 
testing  type  of  activity  and  comes  from  FIPS  11-3,  Guideline:  American 
National  Dictionary  for  Information  Systems  [FIPS11-3]: 

"(1)  tests  to  determine  whether  an  implemented  system  fulfills  its 
requirements.  (2)  see  data  validation  [(1)  a process  used  to 
determine  if  data  are  inaccurate,  incomplete,  or  unreasonable;  the 
process  may  include  format  checks,  completeness  checks,  check  '-'ey 
tests,  reasonableness  checks  and  limit  checks.  (2)  the  checking  of 
data  for  correctness  or  compliance  with  applicable  standards,  rules, 
and  conventions.]" 

Since  there  is  so  much  double  entendre  in  using  the  word  "validation,"  one 
must  define  the  use  applicable  to  the  document.  This  report  is  primarily 
addressing  the  validation  activities  associated  with  standards  and 
conformance  testing;  therefore,  the  two  relevant  definitions,  which  apply 
to  these  testing  phases,  have  been  defined  in  the  glossary. 

Verification. 

DOD-STD-2167A: 

"the  process  of  evaluating  the  products  of  a given  software 
development  activity  to  determine  correctness  and  consistency  with 
respect  to  the  products  and  standards  provided  as  input  to  that 
activity"  [2167A], 

and  FIPS  PUB  132: 

"the  process  of  determining  whether  or  not  the  products  of  a given 
phase  of  the  software  development  cycle  fulfill  the  requirements 
established  during  the  previous  phase"  [FIPS132] 

are  consistent  in  referring  to  an  evaluation  process  applied  to  software 
during  its  development  cycle— component  testing. 
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The  only  solution  to  resolve  misunderstanding  is  to  define  the  terms  in 
the  context  of  a document  or  conversation,  using  one  of  the  several 
references  available  for  doing  so.  Appendix  A defines  terminology  as  it 
relates  to  testing  and  this  report.  Acronyms  used  in  this  report  are  also 
enumerated. 

Individual  information  technology  standards  often  have  their  own 
established  testing  terminology  beyond  those  terms  universally  accepted. 
An  example  of  some  of  this  unique  terminology  is  the  "PIGS"  (protocol 
implementation  conformance  statement)  used  in  GOSIP  (Government  Open 
Systems  Interconnection  Profile)  and  STEP  (Standard  for  the  Exchange  of 
Product  Model  Data)  conformance  testing.  The  reader  will  be  introduced  to 
some  of  this  unique  terminology  when  reading  about  the  status  of  several 
conformance  testing  services  described  near  the  end  of  this  report. 
Supportive  definitions  for  such  terms  are  also  provided  in  Appendix  A. 
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III.  BACKGROUND. 


A.  Standardization  and  Implementation  Process  Flow. 

The  explosion  of  technology  standards  in  the  last  decade  has 
brought  the  issue  of  conformance  testing  to  the  fore.  Through 
standards,  we  have  arrived  at  the  dawn  of  the  age  of  Open 
Systems,  where  one  can  employ  configuration  components  from 
disparate  vendors  to  achieve  cost  and  operating  efficiencies. 

Multi-vendor  systems  without  conformance  testing,  however, 
are  a scourge  upon  the  landscape,  as  they  almost  certainly  will 
fail  to  operate  in  a heterogeneous  product  environment. 
Although  standards  have  unlocked  the  door  to  openness,  they 
can  be  ambiguous,  occasionally  irrational  and  imprecise: 
standards  are  almost  always  implemented  in  a manner 
somewhat  unique  from  someone  else's  implementation.  This 
often  leads  to  operational  failure  unless  testing  is  performed 
beforehand  [JC90]. 

Figure  1 is  a modification  of  a diagram  developed  by  the  CALS/CE  Industry 
Steering  Group  Ad  Hoc  Testing  Committee.  This  interpretation  of  their 
figure  presents  an  example  of  the  overall  structure  within  which 
standards  are  developed  (step  1),  tested  (step  2),  implementations  built 
(step  3),  conformance  testing  performed  (step  4),  high  level  acceptance 
testing  done  (step  5),  and  finally,  at  a more  detailed  level,  the  output  data 
sets  of  the  implementations  verified  (step  6). 

There  are  specific  expected  outputs  from  this  diagram  in  order  to  meet 
the  enterprise  user's  requirements.  At  the  culmination  of  step  2,  an 
international,  national,  federal,  or  military  standard  or  specification  is 
created.  Step  3 culminates  in  an  implementation  of  that  standard  in  off- 
the-shelf  supplier  offerings,  and  step  4 qualifies  the  supplier's 
implementation  against  the  standard.  Step  5 is  the  integration  of  the 
supplier's  implementation  into  the  enterprise  user's  installed  base.  Step 
6 checks  data  integrity,  and  concludes  with  acceptable  data  sets. 
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Figure  1:  Meeting  Enterprise  User  Requirements 

^ ^ [TEST2  modifledi 


Step  1 is  typically  performed  by  standards  bodies  (e.g.,  international, 
national,  governmental).  Step  2,  for  the  purposes  of  this  report,  is  the 
shared  province  of  the  CALS  Test  Network  (for  CALS  standards  and 
specifications)  and  the  National  PDES  Testbed  (for  STEP).  Step  3 is 
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accomplished  by  the  supplier.  Oversight  of  step  4 is  the  responsibility  of 
NIST;  however,  actual  conformance  testing  processes  may  be  delegated  to 
other  agents  acting  on  behalf  of  NIST,  or  performed  as  a suppler's 
declaration.  In  step  5,  an  enterprise  user  may  optionally  choose  to 
perform  interoperability  testing  prior  to  the  remaining  high-level 
acceptance  testing  activities,  e.g.,  performance  testing,  robustness 
testing.  Interoperability  as  a first  step  of  acceptance  testing  is 
particularly  important  if  the  enterprise  user  has  a high  investment  in 
legacy  systems.  Steps  5 and  6 are  typically  performed  by  the  enterprise 
users  of  the  standard  implementations  [TEST2].  Although  interoperability 
is  ultimately  the  responsibility  of  the  enterprise  user,  suppliers  may 
optionally  perform  interoperability  assessment  among  themselves.  An 
example  of  such  a practice  occurs  for  GOSIP  implementations,  where  an  on- 
line system  is  available  for  access  by  suppliers. 

In  general,  an  enterprise  user  is  an  organization  or  person  who  builds, 
uses,  maintains,  or  disposes  of  information  generated  from  an 
implementation.  It  is  important  to  recognize  two  different  enterprise 
users  benefiting  from  the  output  of  steps  5 and  6:  the  systems  integrator 
or  a government  agent.  If  the  buying  government  agent  recognizes  and 
approves  the  requirements  which  a systems  integrator  must  meet  when 
performing  system  and  data  acceptance,  then,  although  the  agent's 
acceptance  testing  task  may  not  be  complete,  the  workload  for  evaluation 
is  minimized.  The  CALS  Evaluation  and  integration  Office  (CEIO) 
could  propose  a consistent  method  for  defining  the  requirements 
which  the  systems  integrator  must  meet  when  assessing  the 
supplier's  implementation  for  a potential  government 
acquisition. 

An  introduction  to  the  various  levels  of  standards,  which  play  a role  in 
CALS,  seems  necessary  in  order  to  understand  the  implication  on  testing 
activity. 

International  Organization  for  Standardization  HSOT 

In  1946,  national  standards  organizations  from  25  countries  formed  the 
International  Organization  for  Standardization  (ISO).  The  U.S. 
representative,  American  National  Standards  Institute  (ANSI),  is  the  sole 
U.S.  representative  to  ISO.  ISO  develops,  coordinates,  and  promulgates 
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international  standards  that  facilitate  world  trade,  contribute  to  the 
safety  and  health  of  the  public,  and  protect  the  environment.  The 
standards  cover  all  fields  except  the  electrotechnical,  which  is  the 
responsibility  of  the  International  Electrotechnical  Commission  (lEC) 
[NCGA]. 

International  Electrotechnical  Commission  HECT 

"In  1906,  the  lEC  was  formed  by  national  committees  from  what  is  now  a 
total  of  44  countries.  The  lEC  develops  and  promulgates  electrotechnical 
standards"  [NCGA]. 

Many  of  those  international  standards,  which  CALS  has  adopted  or  is 
considering  (e.g..  Standard  Generalized  Markup  Language  {SGML},  Computer 
Graphics  Metafile  {CGM},  GOSIP,  Database  Language  SQL),  are  developed 
under  joint  ISO/IEC  sponsorship.  (In  some  cases,  the  standards'  initial 
development  began  in  ISO  before  the  creation  of  a joint  ISO/IEC  technical 
committee.)  Although  STEP  is  being  developed  in  an  ISO  Subcommittee, 
one  of  its  working  groups  which  will  address  electrotechnical  aspects  of 
STEP  was  formed  as  a joint  ISO/IEC  activity  in  1991. 

American  National  Standards  Institute  fANSh. 

The  American  National  Standards  Institute  (ANSI)  coordinates  voluntary 
standards  activities  in  the  United  States  and  is  the  agency  that  approves 
standards  as  American  National  Standards.  It  coordinates  and  manages 
U.S.  participation  in  the  work  of  several  nongovernmental  international 
standards  organizations,  including  ISO  and  lEC  [NCGA]. 

Federal  Information  Processing  Standards  fFIPSL 

The  National  Institute  of  Standards  and  Technology  (NIST)  works  through 
voluntary  industry  standards  organizations,  such  as  ISO  and  ANSI,  to 
develop  standards  that  will  meet  the  needs  of  Government  users  and  be 
implemented  in  commercial  off-the-shelf  products.  Standards  that 
promise  sizable  benefits  to  the  Government  are  issued  as  Federal 
Information  Processing  Standards  (FIPS)  [SJK88].  FIPS  are  developed  by 
NIST  and  issued  by  the  Secretary  of  Commerce  to  serve  as  legislative  and 
executive  mandates  for  improving  the  use  and  management  of  computers 
and  ADP  systems  in  the  federal  government.  The  goals  of  the  FIPS 
program  are  to: 
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"Improve  the  life-cycle  efficiency  and  effectiveness  of  federal 
information  technology  resources; 

facilitate  the  competitive  and  economic  procurement  of  systems, 
components,  and  services; 

improve  the  portability  of  data,  software,  and  technical  skills 
across  systems; 

protect  systems  and  networks  against  unauthorized  access, 
manipulation,  abuse,  and  protect  information  from  unauthorized 
modification  or  disclosure; 

reduce  waste,  errors,  and  unnecessary  duplication  in  the  application 
and  use  of  systems;  and 

increase  the  productivity  of  the  federal  work  force"  [FIPS91]. 

Military  Standards  and  Specifications. 

Military  standards  (MILSTDs)  establish  engineering  and  technical 
requirements  for  processes,  procedures,  practices,  and  methods  that  have 
been  adopted  as  a standard.  Military  specifications  (MILSPECs)  are 
prepared  specifically  to  support  defense  acquisition.  They  are  intended  to 
clearly  and  accurately  define  essential  technical  requirements  as  well  as 
define  that  those  procedures  necessary  to  determine  requirements  have 
been  met  [DOD  4120.3M].  Specific  to  CALS,  the  purpose  of  MILSTD-1840  is 
to  standardize  the  format  and  information  structures  of  digital  data  files 
used  for  the  transfer  and  archival  storage  of  digital  technical  information 
[1840].  The  MILSPECs  28000,  28001,  28002,  and  28003  identify 
requirements  for  specific  applications  using  the  Initial  Graphics  Exchange 
Specification  (IGES),  Standard  Generalized  Markup  Language  (SGML), 
Raster,  and  the  Computer  Graphics  Metafile  (CGM),  respectively. 

"Tailoring"  standards  for  unique  requirements,  while  still  maintaining 
technical  attributes  from  the  standard,  is  a potential  occurrence,  given 
the  generation  of  standards  at  the  international,  national,  federal,  and 
defense  levels.  For  example,  given  the  existence  of  an  ISO  standard,  ANSI 
can: 


Accept  it  in  total. 

Delete  portions  unnecessary  to  achieve  United  States 
objectives. 


Add  additional  functionality  not  included  in  the  original 
standard. 

Make  the  standard  more  restrictive  for  the  United  States  than 
what  was  approved  internationally  by  limiting  options 
specified  in  the  standard. 

Any  action  other  than  option  1)  would  be  carried  out  only  'o  tailor  the 
standard  more  closely  to  United  States  needs.  These  same  options  would 
also  be  considered  when  developing  a FIPS  in  order  to  best  accommodate 
the  requirements  specific  to  the  federal  government.  Then,  in  turn,  the 
CEIO  may  apply  further  tailoring  when  issuing  military  specifications  to 
address  specific  needs  of  the  Department  of  Defense  (DoD). 

B.  Types  of  Testing. 

Figure  2 selects  testing  activities  of  Figure  1 which  are  general  to  all 
CALS  enterprise  user  environments,  emphasizing  that  testing  activities 
occur  as  a continuum.  Standards  testing  (which  includes  military 
specification  testing)  determines  whether  the  national,  international,  or 
military  standards  (and  specifications)  are  viable  and  implementable. 

Component  testing  is  the  routine  checking  performed  during  the 
development  cycle  of  an  implementation.  Conformance  testing  and 
acceptance  testing  ensure  the  implementation  meets  the  standard  and  also 
meets  the  enterprise  user's  requirements  respectively.  At  all  levels  of 
testing,  there  is  preparatory  work  that  occurs  prior  to  the  actual  testing 
activities.  With  the  exception  of  acceptance  testing,  the  degree  of  testing 
activity  diminishes  and  levels  off  as  the  standards,  military 
specifications,  and  products  become  more  stable.  Since  acceptance 
testing  is  tailored  to  meet  each  enterprise  user's  requirements,  the  level 
of  activity  remains  somewhat  constant. 

Standards  testing  has  not  traditionally  occurred  during  the  standard's 
development  process.  The  decision  to  conduct  such  testing  is  based  on  the 
level  of  available  technology  to  support  the  standard  during  its 
development.  Historically,  standards  have  been  produced  after  the 


technology  is  established  and  implementations  are  developed.  For  the 
development  of  STEP,  a different  approach  was  chosen.  Building  on 
research  and  development  (R&D),  but  not  on  vast  implementation 
experience,  STEP  is  being  designed  through  the  visions  of  many 
individuals.  Without  reference  implementations  developed  during 
standards  testing,  which  may  be  used  to  pass  quality  judgement  on  the 
concepts,  the  result  may  be  a product  data  exchange  standard  that  is  (1) 
not  implementable  or,  if  implementable,  (2)  does  not  solve  the  functional 
requirements  the  enterprise  user  community  initially  expressed.  Hence, 
"validating"  the  standard  is  necessary  prior  to  its  adoption. 

The  CALS  military  specification  testing  was  founded  on  a similar  need. 
These  specifications  were  written  to  minimize  the  flexibility  usually 
found  in  consensus-built  national  or  international  standards,  and  add  any 
additional  requirements  specific  to  the  CALS  initiative.  No 
implementations  of  these  military  specifications  existed  prior  to  their 
adoption,  and  the  concept  of  requiring  computer  standards  for  the 
exchange  of  weapon  system  data  was  relatively  new.  Given  this 


Figure  2:  TESTING  ACTIVITY  CONTINUUM 
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environment,  it  was  necessary  to  establish  a CALS  testing  program  to 
assess  the  viability  of  the  military  specifications  themselves.  Although 
the  mission  of  standards  testing  and  military  specification  testing  is 
similar,  the  distinction  in  Figure  2 is  due  to  timing  of  each  testing 
activity  relative  to  the  adoption  of  a national  or  international  standard. 


Component  testing  is  conducted  internally  by  a supplier  against  his 
implementation  of  a standard  during  the  product  development  cycle.  It  is 
testing  that  has  occurred  since  software  development  began  and  is  not 


Figure  3:  CONFORMANCE  TESTING  PROGRAM  MODEL 

directly  influenced  by  the  CALS  initiative.  It  is  mentioned  here  only  to 
acknowledge  the  continued  importance  of  such  testing. 
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Conformance  testing  is  the  testing  of  a candidate  product  for  the 
existence  of  characteristics  required  by  a standard.  Its  primary  activity 
is  to  ensure  specified  behavior  of  implementations.  Additional  benefits 
include;  clarifying  the  standard  for  guiding  future  implementation, 
producing  a feedback  loop  to  the  standards-making  bodies  for 
improvements  to  the  standard,  encouraging  commercial  development  by 
supporting  a baseline  for  commonality  in  all  products,  and  providing 
greater  confidence  on  the  part  of  the  potential  enterprise  user. 
Conformance-tested  implementations  increase  the  probability  these  same 
implementations  will  be  able  to  interoperate,  but  provides  no  guarantee. 
Figure  3 portrays  the  activities  of  conformance  testing. 

In  the  formal  conformance  testing  process,  the  client  is  the  organization 
or  individual  seeking  recognition  that  a product  complies  with  the 
standard.  Upon  completing  conformance  testing,  the  client  obtains  a 
conformance  test  report.  This  test  report  may  enhance  a commercial 
client's  selling  power  to  bid  on  a government  contract  or  show  a potential 
system's  integrator  or  commercial  user  the  product  has  been  tested  under 
a controlled  environment  by  an  unbiased  testing  laboratory  using  approved 
test  methods.  This  formal  process  improves  the  competitive  edge  for  the 
client  against  those  suppliers  who  have  not  undergone  the  same  process. 
It  is  also  important  for  a government  developer  to  undergo  conformance 
testing  of  its  implementations  to  ensure  conformity  prior  to  use. 

An  alternative  to  this  formal  process  is  the  supplier's  declaration.  This  is 
where  the  supplier  performs  its  own  testing,  generates  a test  report,  and 
makes  a "declaration"  of  conformance  for  a given  implementation  against 
a given  standard. 

Although  conformance  testing  provides  a means  to  evaluate  syntactic  and 
semantic  alignment  of  an  implementation  to  a given  standard,  it  does  not 
measure  fitness  for  use  in  a particular  environment.  The  enterprise  user 
is  interested  in  a systems  approach.  Enterprise  users  must  provide  their 
own  way  to  measure  robustness,  performance,  interoperability,  or  data 
integrity  of  an  implementation  and  the  system  under  which  it  operates. 
These  activities  are  known  as  acceptance  testing.  Basically,  the  burden 
falls  to  the  enterprise  user  to  perform  acceptance  testing,  since  it  is 
defined  for  a particular  functional  requirement  in  the  context  of  a 


particular  operation.  Acceptance  testing  might  be  applied  to  applications 
used  to  capture,  manipulate,  and  manage  data  which  is  to  be  delivered  or 
made  available  to  customers  for  their  use. 

"Acceptance"  is  defined  in  the  Federal  Acquisition  Regulation  paragraph 
46.101  as:  "the  act  of  an  authorized  representative  of  the  government  by 
which  the  government,  for  itself  or  as  agent  of  another,  assumes 
ownership  of  existing  identified  supplies  tendered  or  approves  specific 
services  rendered  as  partial  or  complete  performance  of  the  contract." 
Modified  to  apply  to  this  report,  acceptance  testing  is  the  act  of  an 
authorized  agent  of  the  government,  whereby  the  agent  assumes  ownership 
of  data  products  or  approves  specified  data  services  rendered  for  the 
contract. 

Two  major  factors  influence  the  type  and  degree  of  acceptance  testing 
appropriate  in  a CALS  environment:  (1)  the  volume  of  information  and 
frequency  of  interchange  and  (2)  the  inherent  quality  improvements  using 
computers,  computer  networks,  and  software-controlled  quality.  When 
data  products,  data  interchange,  and  on-line  access  services  have 
undergone  conformance  testing  to  CALS  military  specifications, 
acceptance  testing  is  greatly  simplified  [ISG90]. 

When  one  discusses  the  current  CALS  military  standards, 
interoperability  testing  should  be  part  of  acceptance  testing 
since  the  enterprise  user  has  the  ultimate  responsibility  for 
interoperability.  This  will  ensure  that  any  introduction  of  new 
hardware  and  software  can  communicate  with  current  legacy  systems. 
Interoperability  testing  is  used  to  evaluate  the  effectiveness  and 
usability  of  data  exchange  mechanisms  with  respect  to  an  enterprise 
user's  environment  and  requirements.  The  effectiveness  of  the  data 
transfer  may  be  influenced  by  combined  interactions  of  the  legacy  system 
and  enterprise  user's  practices  [ITM91]. 

C.  Existing  Directives  and  Guidelines. 

The  continuum  of  testing  activities  is  not  new  to  the  government  or  the 
commercial  sector.  Many  policies  have  been  generated,  standards  written, 
and  guidelines  prepared  to  better  facilitate  smooth  testing  operations. 


Appendix  B is  a list  of  directives  and  guidelines  of  general  value  to  the 
testing  continuum,  as  well  as  those  of  specific  value  to  the  CALS 
initiative.  Although  the  list  is  not  exhaustive,  it  should  impress  upon  the 
reader  the  many  interested  domains  that  have  given  thought  to  testing 
activities. 
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IV.  TESTING  ACTIVITIES. 


A.  General  Description  of  Primary  Participants. 

Before  describing  a particular  testing  activity  performed  by  a specific 
organization,  it  seems  appropriate  to  introduce  the  three  primary 
participants  within  the  CALS  community:  the  CALS  Test  Network  (CTN), 
the  National  Institute  of  Standards  and  Technology  (NIST),  and  the 
CALS/CE  Industry  Steering  Group  (ISG).  All  have  some  level  of  activity 
across  the  breadth  of  testing  under  the  CALS  initiative. 

1.  CALS  Test  Network 


The  CALS  Test  Network  (CTN)  is  a confederation  of  over  300  industry  and 
30  government  organizations  that  have  agreed  to  evaluate  and 
demonstrate  the  interchange  and  functional  use  of  digital  technical 
information  using  CALS  standards.  This  is  accomplished  through  a 
collaborative  multi-service  effort.  Test  beds  in  support  of  the  CALS  Test 
Network  Office  have  been  established  by  each  of  the  DoD  Services,  Defense 


Opticil  Dlik  Drive 


Figure  4:  Representative  CTN  Test  Bed  Hardware 

ConHguration  [1ITE91] 


Logistics  Agency  (DLA),  and  Lawrence  Livermore  National  Laboratory. 
Figure  4 is  a representative  configuration  of  a typical  test  bed  used  to 
support  CTN  activities  [CTN91]. 

These  test  beds  and  the  activities  they  perform  are  funded  and  managed  by 
the  CEIO.  The  Air  Force  serves  as  the  Operations  Manager  of  the  CTN  for 
the  CEIO,  and  oversees  the  day-to-day  technical  and  administrative 
operations  of  the  various  test  beds. 

2.  National  Institute  of  Standards  and  Technology. 

Created  in  1901  as  the  National  Bureau  of  Standards  and  renamed  in  1988, 
the  National  Institute  of  Standards  and  Technology  (NIST)  works  to 
strengthen  U.S.  industry's  international  competitiveness,  advance  science, 
and  improve  public  health,  safety,  and  the  environment.  NIST  conducts 
science  and  engineering  research  in  commercially  important  fields  such  as 
advanced  materials,  information  systems,  biotechnology,  optoelectronics, 
computer-integrated  manufacturing,  and  sensor  technology. 

NIST's  laboratory  research  is  designed  to  support  development  of  critical 
emerging  technologies  and  the  new  measurement  methods  and  standards 
necessary  to  make  them  commercially  viable  [NIST91].  There  are  several 
functional  operations  within  NIST  which  contribute  to  some  aspect  of 
testing  activity  relevant  to  the  CALS  initiative.  The  CEIO  has  sponsored 
standards  testing  and  conformance  testing  activities  within  several  NIST 
laboratories. 

3.  CALS/CE  Industry  Steering  Group. 

The  Industry  Steering  Group  (ISG)  provides  the  National  Security 
Industrial  Association  (NSIA)  thrust  in  CALS  and  concurrent  engineering. 
The  CALS/CE  ISG  acts  as  the  formal  industry  interface  with  the 
Department  of  Defense  CEIO.  Several  hundred  volunteers  participate  in  the 
more  than  30  committees.  The  ISG  also  coordinates  the  efforts  of  20 
industry  associations  to  respond  to  DoD  direction  for  the  CALS  effort.  The 
CALS/CE  Industry  Steering  Group  (ISG)  has  recognized  the  necessity  of 
testing  for  some  time.  Two  committees  have  been  created  to  assess 
requirements  and  make  recommendations. 
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B.  Standards  Testing  Activities. 


1.  National  PDES  Testbed  (NPT). 

The  goal  of  the  National  PDES  Testbed  (NPT),  located  at  the  National 
Institute  of  Standards  and  Technology  (NIST),  is  to  provide  technical 
leadership  and  a testing-based  foundation  for  the  complete  development 
of  STEP.  Meeting  the  objectives  associated  with  several  processes  is 
necessary  to  validate  the  viability  and  implementability  of  the 
International  Organization  for  Standardization  (ISO)  Committee  Draft  (CD) 
for  STEP: 

Model  Scoping  and  Construction.  Identify  and  model  an  application  use  of 
the  draft  STEP  specifications. 

Test  Definition  Tool.  Define  test  scenarios  for  evaluating  STEP  models. 

Test  Case  Generation  Tool.  Convert  real  world  product  data  into  STEP 
structures. 

Test  Execution  and  Evaluation  Tool.  Run  test,  analyze  results,  and  produce 
final  reports  [4417-90]. 

Of  the  many  technical  threads  pursued  within  the  NPT,  those  related 
specifically  to  CALS  standards  testing  include: 

Testbed  Initiation  Activities.  Establishing  the  operational  testbed 
facility,  coordinating  efforts  with  outside  organizations,  performing 
initial  technical  studies,  developing  prototype  systems,  and 
performing  preliminary  testing  of  the  draft  STEP.  (These  initiation 
activities  were  completed  in  fiscal  year  1990.) 

Validation  Test  System.  Developing  a system  for  testing  and 
evaluating  the  application  protocols  which  are  defined  as 
standardized  parts  of  STEP.  (NIST  released  a version  at  the  end  of 
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March  1992  which  conformed  to  the  April  1991  release  of  Express. 3 
A tool,  which  allows  population  of  an  application  test  model  written 
in  Express,  is  currently  under  alpha  test  mode. 

Application  Protocol  (AP)  Specification  and  Validation.  Specifying 
and  validating  at  least  one  application  protocol  defined  in  STEP 
[4438-90].  (The  NPT  is  refining  the  process  used  previously,  and 
recognized  the  need  for  more  stable  resource  models.  The  process  is 
being  aligned  with  the  current  AP  Development  Guidelines.  An  AP 
validation  workshop  was  held  at  the  April  1992  joint  IGES/PDES 
Organization  (IPO)  and  ISO  TC184/SC4  Working  Groups'  meeting 
which  brought  consensus  on  several  issues  concerning  this  process 
of  validation  and  the  AP  development  process.) 

2.  CALS  Test  Network  (CTN). 

The  standards  testing  operation  of  the  CTN  covers:  testing  and 
demonstrating  access  and  exchange  of  digital  technical  data  in  actual  use 
conditions;  developing  reference  test  data  and  criteria  for  test  planning; 
and  developing  procedures  to  be  followed  by  government  and  industry  test 
facilities  [CTN90].  Some  of  the  initial  target  testing  areas  include: 
technical  publications  (text  and  graphics),  engineering  data  (engineering 
drawings  and  computer-aided  design  (CAD)  files),  support  data  (on-line 
access  to  logistic  support  analysis  (ISA)  data),  digital  data  protection  and 
security,  digital  data  configuration  management,  digital  data  acceptance 
procedures,  and  CALS  for  small  business  [CTN91]. 

C.  Conformance  Testing  Activities. 

1.  CALS  Test  Network. 

The  perception  of  the  CTN  by  many  is  that  any  one  of  the  identified  CTN 
test  beds  performs  conformance  testing  for  the  participating  suppliers. 
In  fact,  the  CTN  is  chartered  to  affirm  CALS-compliance  only  for 


3 Express  is  a specification  language  for  capturing  structural  and 
semantic  aspects  of  the  STEP  information  model.  It  will  become  a 
standardized  part  (ISO  10303-11)  of  the  STEP  suite. 
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DoD-owned  systems  [JUN91].  The  CALS  Test  Network  has  been 
requested,  on  occasion,  to  evaluate  implementations  of  CALS 
specifications  which  are  already  procured  under  contract  by  one  of  the 
services  or  DLA.  Their  first  activity  of  this  nature  was  to  ascertain  CALS- 
compliance  for  the  raster  compression  software  (Type  I)  on  the  DSREDS, 
EDCARS,  and  EDMICS  systems. 

2.  FIPS  and  Military  Specification  Conformance  Testing. 

Computer  users  and  the  computer  equipment,  software,  and  services 
industries  receive  support  from  NIST  in  the  form  of  standards  and 
technical  methods.  These  standards  and  technical  methods  are  applied  to 
advance  the  effective  use  of  computers  and  related  telecommunications 
equipment.  One  aspect  of  this  support  is  the  issuing  of  Federal 
Information  Processing  Standards  Publications  (FIPS  PUBS).  NIST  has 
responsibility  for  providing  conformance  testing  programs  for  FIPS  PUBS 
where  a need  has  been  identified  and  resources  are  available.  Under  the 
CALS  initiative,  the  conformance  testing  responsibilities  of 
NIST  have  been  expanded  to  include  conformance  testing  against 
the  CALS  military  specifications. 

NIST's  approach  to  CALS  conformance  testing  has  been:  (1)  establish  a 
conformance  testing  service  for  the  FIPS  if  warranted  and  (2)  adapt  the 
FIPS  conformance  testing  service  to  meet  CALS  requirements  as  specified 
in  the  military  specification  28000  series.  The  formal  policies  and 
procedures  to  define  a conformance  testing  service  for  either  a FIPS  or  a 
military  specification  are  similar. 

With  the  exception  of  Mil-D-28000,  based  on  the  Initial  Graphics  Exchange 
Specification  (IGES),  the  CALS  military  specifications  require  FIPS 
compliance.  A proposed  FIPS  for  IGES  is  now  under  consideration.  The 
FIPS  for  CGM  (FIPS  PUB  128)  is  being  modified  to  include  the  full 
functionality  of  MIL-D-28003.  Once  this  process  is  complete,  FIPS  CGM 
conformance  testing  and  MIL-D-28003  conformance  testing  will  be 
synonymous. 
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3.  National  Voluntary  Laboratory  Accreditation  Program  (NVLAP). 

Independent  of  other  NIST  activities,  NVLAP  is  a formal  accreditation 
program  which  qualifies  first,  second,  and  third  party  testing  laboratories 
for  performing  specific  tests  or  types  of  conformance  tests.  Specifically, 
it: 


Provides  national  recognition  for  competent  testing  laboratories. 
Provides  testing  laboratory  management  with  a quality  assi’^'ance 
check  of  the  performance  of  their  laboratories. 

Identifies  competent  testing  laboratories  for  use  by  regulatory 
agencies,  purchasing  authorities,  and  product  certification  systems. 
Provides  testing  laboratories  with  guidance  from  technical  experts 
to  aid  in  reaching  a higher  level  of  performance  resulting  in 
generating  improved  engineering  and  product  information. 

NVLAP  is  comprised  of  a series  of  laboratory  accreditation 
programs  which  are  established  on  the  basis  of  requests  and 
demonstrated  need.  The  specific  test  methods,  types  of  test  methods, 
products,  services,  or  standards  to  be  included  in  an  accreditation 
program  must  be  requested.  The  Director  of  NIST  does  not  unilaterally 
propose  or  decide  the  scope  of  a laboratory  accreditation  program. 
Communication  with  other  laboratory  accreditation  systems  is  promoted 
to  encourage  development  of  common  criteria  and  approaches,  and  promote 
acceptance  of  test  data  produced  by  the  accredited  laboratories.  NVLAP  is 
intended  to  be  compatible  with  and  recognized  by  domestic,  foreign,  and 
international  laboratory  accreditation  systems  which  enhances  the 
universal  acceptance  of  test  data  produced  by  NVLAP-accredited 
laboratories  [4493-90]. 

There  is  a cost  to  the  testing  laboratory  desiring  accreditation.  Table  I 
shows  some  examples  of  accreditation  costs  associated  with  other 
computer  standards,  as  well  as  an  example  for  a military  standard 
accreditation  program. 

Criteria  for  judging  a testing  laboratory's  competence  include:  evaluating 
the  laboratory's  quality  system,  staff,  facilities,  equipment,  test  methods 
and  procedures,  records,  and  test  reports.  The  actual  accreditation  of  the 
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testing  laboratory  includes  an  on-site  assessment  by  a team  technically 
competent  in  the  standard  and  testing  tools  for  which  the  laboratory  is 
being  accredited.  The  testing  laboratory's  accreditation  is  based  on  the  on- 
site assessment  reports,  actions  taken  by  the  testing  laboratory  to 
correct  deficiencies,  results  of  proficiency  testing,  and  information  from 
any  monitoring  visits  which  may  have  been  performed. 


Fee  Schedule  (effective  10/1/91) 


PROGRAM 

ANNUAE 

AD/TECH 

SUPP.  FEE(l) 

LNITFAE 

APPLIC. 

FEE(2) 

DN-S'lTE 

ASSESS. 

FEE(3) 

PROFICIENCY 

TESTING 

FEE 

TEST 

METHOD 

FEE  (each) 

Computer/GOSIP 

$5,600 

$500 

$2,100 

$0 

$300 

Computer/High 

Level  Protocols 

3,500 

500 

1,300 

100 

150 

Computer/POSIX 

3,600 

500 

3,000 

0 

150 

Computer/X.25- 
B lacker 

3,500 

500 

1,300 

100 

150 

ECT/MILSTD-462 

3,550 

500 

3,000 

200 

30 

(1)  The  Administrative/Technical  Support  Fee  is  assessed  annually,  regardless  of  a laboratory's 
accreditation  status. 

(2)  One  time  per  program  only.  This  fee  not  paid  if  a renewal  application. 

(3)  The  On-Site  Assessment  Fee  is  due  every  other  year.  This  fee  paid  only  in  the  year  in  which  notification  is 
received  that  an  on-site  assessment  will  be  performed. 

Table  I:  NVLAP  Fee  Structure  [1 144K] 


4.  National  PDES  Testbed  (NPT) . 

Besides  the  standards  testing  activities  mentioned  above,  the  National 
PDES  Testbed  has  also  been  funded  by  the  CEIO  in  the  past,  to  actively 
participate  in  the  development  of  the  STEP  conformance  testing  standards 
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within  the  ISO  community.^  Some  of  the  services  provided  in  this  area 
included:  representing  U.S.  and  CEIO  interests  in  the  international  STEP 
development  process,  coordinating  the  U.S.  position  on  the  STEP 
conformance  testing  part  ballot,  providing  a development  plan  for 
building  and  institutionalizing  a conformance  testing  system  and 
framework  for  validating  implementations  of  STEP  application  protocols 
[4641-91],  and  assessing  the  general  requirements  of  STEP  conformance 
testing  [4743-92]. 

5.  CALS/CE  Industry  Steering  Group  Committee . 

An  Ad  Hoc  Testing  Committee  was  formed  to  assess  the  status  of  testing 
activity  and  identify  the  voids.  Ultimately,  the  Committee  was  concerned 
with  ensuring  whether  the  facilities  existed  to  guarantee  commercial 
availability  of  CALS-conforming  products.  Representatives  from  industry, 
NIST,  CTN,  and  Sandia  Laboratories  participated  in  evaluating  the  level  of 
testing  coverage.  The  ad  hoc  committee  culminated  its  initial  activity  by 
delivering  two  reports  to  the  CALS  Offices  in  the  fall  of  1990,  and  spring 
of  1991.  In  the  spring  of  1992,  the  Committee  reconvened  to  assess 
where  it  left  off  and  draft  the  final  report  of  recommendations  to  the 
CEIO.  These  recommendations  include: 

Create  FIPS  for  all  CALS  standards  and  military  specifications. 

Require  NIST  to  develop  third-party  certification  procedures. 

Apply  certification  procedures  to  NIST  and  third  party  product 

validation  laboratories. 


4 The  Class  30  series  on  conformance  testing  methodology  and 
framework  will  eventually  be  part  of  the  STEP  suite:  10303-31:  general 
concepts;  10303-32:  the  requirements  on  testing  laboratories  and 
clients:  10303-33:  structure  and  use  of  abstract  test  suites;  and  10303- 
34:  abstract  test  methods. 

5 During  the  calendar  year  1991,  the  "CALS  Office"  was  changed  to 
the  "CALS  Evaluation  and  Integration  Office"  (CEIO). 
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Place  all  software  vendor  offerings  that  have  completed  the 
certification  process  on  the  Validated  Products  List  (VPL)6  [1SG92]. 

D.  Acceptance  Testing  Activities. 

1.  CALS  Test  Network. 

As  an  overall  tasking,  the  CTN  Office  is  responsible  for 
government  user  application,  interoperability,  and  other  related 
testing.  All  of  the  test  beds  perform  interoperability  testing  to  the 
level  of  the  test  bed  capability,  i.e.,  test  bed  system  hardware  and 
software  available.  The  Army  test  bed  has  been  designated  the  lead 
service  specific  to  data  acceptance  testing.  Particularly,  CTN  Test  Report 
91-028  presents  models  that  define  the  entities  and  attributes  related  to 
the  acceptance  of  CALS-conforming  digital  data  for  technical  manuals. 
These  models  are  expandable  as  the  CALS  specifications  mature. 

2.  CALS/CE  Industry  Steering  Group  Committee. 

The  CALS/CE  ISG  Acceptance  Testing  Committee  developed  a report  for 
industry  and  the  CALS  Office  to: 

Recommend  approaches  which  may  be  applied  to  assure  successful 
data  exchange  for  near  term  CALS  deliverables. 

Recommend  approaches  which  may  be  applied  to  ensure  successful 
access  to  and  use  of  information  which  resides  within  the 
Contractor's  Integrated  Technical  Information  Service  (CITIS)  for  its 
intended  purposes. 

Provide,  for  any  given  type  of  CALS  deliverable,  a framework  for 
determining:  what  the  evaluation  method  options  are,  what  the 
acceptance  criteria  should  be,  what  constitutes  contractual 


6 The  Validated  Products  List  is  an  internationally-recognized 
document  which  publishes  results  for  several  conformance  testing 
services.  It  is  available  through  the  National  Technical  Information 
Service:  subscriptions,  phone:  703/487-4630;  individual  copies,  phone: 
703/487-4650;  ordering  number  PB91-937300. 
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delivery,  and  when  and  how  formal  acceptance  should  be 
accomplished  [ISG90]. 

The  Acceptance  Testing  Committee  submitted  this  report  to  the  CALS 
Office  in  July  1990.  Generally  speaking,  this  report  tells  how  an 
enterprise  user  knows  when  data  has  been  received,  not  whether  or  not  the 
data  received  is  good. 
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V.  TESTING  ISSUES  ASSOCIATED  WITH  CALS 
INITIATIVE. 

The  following  issues  have  been  raised  and  were  highlighted  in  the  previous 
text  (sections  I through  IV),  and  are  gathered  here  for  further  elaboration. 
After  each  issue  is  raised,  alternatives  are  offered,  the  pros  and  cons  for 
each  alternative  listed,  and  a recommendation  made.  The  author  is 
providing  only  recommendations:  The  CEIO  has  to  consider  DoD  flagship 
program  priorities,  adaptation  with  other  defense  initiatives  such  as  CIM 
(Corporate  Information  Management),  and  budgetary  constraints  when 
determining  the  most  appropriate  alternative.  In  some  cases  more  than 
one  alternative  may  be  selected  since  the  alternatives  are  not  always 
mutually  exclusive.  The  order  of  presentation  is  not  meant  to  reflect  a 
priority. 

A.  Should  the  CEIO  Allow  the  Systems  Integrator  to 
Perform  Acceptance  Testing? 

The  CEIO  could  propose  a consistent  method  for  defining  the 
requirements  which  the  systems  integrator  must  meet  when 
assessing  the  supplier's  implementation  for  a potential 
government  acquisition.  Since  acceptance  testing  is  a responsibility 
associated  with  each  procurement,  it  can  often  be  very  expensive. 

Issue.  Whether  the  CEIO  should  recognize  any  acceptance  testing 
performed  by  the  systems  integrator  prior  to  government  procurement. 

Alternative  1 . Prior  to  obtaining  a system,  the  government  user  performs 
full  acceptance  testing. 

Pros. 

Ensures  the  highest  level  of  confidence  for  the  government  user  that 
the  system  meets  the  requirements. 


-Cons. 

Expensive  and  repetitive  for  each  and  every  government  user, 
especially  when  buying  systems  which  meet  similar  requirements  of 
other  earlier  procurements. 
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Most  likely  duplicating  the  expense  and  effort  previously  performed 
(at  least  in  part)  by  the  systems  integrator. 

Government  staff  and  equipment  must  be  brought  up-to-speed  to 
perform  the  acceptance  testing  tasks. 

Adds  overhead  cost  to  overall  procurement. 

May  not  be  legal  prior  to  contract  award  under  the  procurement 
process. 

Alternative  2.  The  government  user  provides  for  some  level  of  acceptance 
testing  being  performed  by  the  systems  integrator  without  consideration 
of  due  process. 

Pros. 

Diminishes  workload  and  cost  requirement  on  government  user  to 
perform  acceptance  testing. 

Requires  less  in-house  expertise  and  equipment  for  performing  the 
remainder  of  the  acceptance  testing  activities. 


Cons. 

Provides  no  consistent  means  of  measurement  from  systems 
integrator  to  systems  integrator. 

Requires  level  of  trust  that  systems  integrator  performed  the 
required  steps  to  meet  the  procurement  criteria. 

Each  government  user  must  determine  what  can  be  performed  by  the 
systems  integrator  and  what  must  be  left  for  the  government  user  to 
do. 

Alternative  3.  The  government  user  applies  predefined  requirements 
which  the  systems  integrator  applies  in-house,  and  the  government  user 
accepts  the  results. 

Pros. 

Diminishes  workload  and  cost  requirement  on  government  user  to 
perform  acceptance  testing. 

Requires  less  in-house  expertise  and  equipment  for  performing  the 
remainder  of  the  acceptance  testing  activities. 

Provides  a consistent  means  of  measurement  across  systems 
integrators. 
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Provides  a consistent  means  of  measurement  across  procurements. 
Diminishes  workload  of  government  user  to  establish  means  of 
measurement  for  each  procurement. 

Leverages  what  is  usually  performed  by  the  systems  integrator 
anyway. 


Cons. 

Government  user  still  left  with  some  acceptance  testing. 

A predefined  measurement  of  the  systems  integrator  process  may 
have  to  be  tailored  for  each  systems  integrator  or  each  standard. 

The  systems  integrator  may  believe  the  government  has  no  business 
assessing  the  way  it  conducts  business  in-house. 

Recommendation.  Alternative  3:  The  government  user  applies 

predefined  requirements  which  the  systems  integrator  applies  in-house, 
and  the  government  user  accepts  the  results. 

Support  of  such  a recommendation  would  require  the  development  of  the 
requirements  for  this  level  of  acceptance  testing.  Time  and  cost  could  be 
reduced  if  some  of  the  acceptance  tasks  were  performed  by  the  systems 
integrator  prior  to  the  government  user's  assessment.  There  are  too  many 
unique  requirements  associated  with  any  given  procurement  to  develop  one 
grand  scheme  for  acceptance  testing.  Developing  a definition  of  the 
requirements  would  create  a controlled  yet  flexible  environment.  The  CTN 
staff  would  be  appropriate  resources  to  assist  the  CEIO  to  develop  such 
requirements  since  they  have  a combination  of  technical  expertise  in  the 
testing  arena,  as  well  as  an  understanding  of  DoD  requirements. 

B.  How  Should  CEIO  STEP  Validation7  Funds  be 
Apportioned? 

Develop  a system  for  testing  and  evaluating  the  application 
protocols  which  are  being  defined  as  parts  of  STEP.  Should  the 
CEIO  invest  in  a system  for  testing  and  evaluating  the  STEP  application 
protocols  as  they  are  being  drafted?  Is  such  a testing  system  necessary? 


7 Within  the  STEP  community,  standards  testing,  as  defined  in  this 
report,  is  commonly  referred  to  as  STEP  validation. 
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Why  evaluate  the  application  protocols  prior  to  standardization?  Why 
should  NIST  be  the  host  site  of  the  testing  system  and  the  resources? 

Issue.  Is  it  cost-effective  for  the  CEIO  to  fund  a validation  testing 
system,  and  if  so,  where  should  such  a system  be  hosted? 

Alternative  1.  Develop  a Validation  Testing  System  to  test  the  viability 
of  the  STEP  draft  specifications  (e.g.,  application  protocols). 

Pros. 

Since  a reference  implementation  is  built  as  part  of  validation,  it 
ensures  the  standard  is  implementable. 

Early  testing  builds  enterprise  user  and  supplier  confidence  in  the 
standard. 

Proactive  U.S.  participation  in  testing  the  standard  strongly 
influences  the  content  of  the  standard:  therefore,  the  alternative 
helps  to  meet  CALS  requirements. 

Provides  tools  for  potential  use  in  component  testing,  conformance 
testing,  and  acceptance  testing  of  STEP  AP  implementations. 
Validation  testing  system  hardware,  software,  and  environment  have 
long-term  use  even  after  the  initial  release  of  STEP  is  published 
(uses  include:  AP  development  and  testing,  conformance  testing,  and 
acceptance  testing). 


Cons. 

Such  a system  is  potentially  an  expensive  CEIO  investment,  since 
current  investment  levels  are  based  on  the  projected  utility  of  the 
standard  to  meet  DoD  requirements. 

Alternative  2.  If  a Validation  Testing  System  to  test  the  viability  of  the 
STEP  draft  specifications  is  developed,  NIST  should  be  the  host  site. 

Pros. 

NIST  has  no  vested  interest  in  the  requirements  STEP  supports; 
therefore,  NIST  can  offer  technical  guidance  to  help  STEP  meet  CALS 
requirements. 
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Early  participation  in  the  standardization  process  increases  NIST 
understanding  of  the  technologies  for  global  federal  agency 
application. 

NIST  can  better  harmonize  DoD's  requirements  and  priorities  with 
other  federal  agencies  and  the  international  community. 

NIST  has  strongly  established  relationships  with  other  STEP- 
supporting  organizations,  e.g.,  PDES,  Inc. 

The  physical  location  of  the  National  PDES  Testbed  is  centrally 
located  along  with  the  office  of  the  National  Product  Data  Exchan?^^ 
Initiative. 


Cons. 

STEP  expertise  is  not  being  developed  within  DoD. 

Hardware  and  software  investment  is  not  at  a DoD  site. 

NIST  may  not  understand  DoD  requirements  as  well  as  DoD. 

Alternative  3.  Instead  of  the  CEIO  investing  in  hardware,  software,  and 
test  bed  environmental  support  resources  for  the  development  of  STEP,  it 
should  invest  in  additional  technical  human  resources. 

Pros. 

Funding  levels  can  more  easily  fluctuate  because  not  impacted  by 
hardware,  software,  site  maintenance  costs. 

Provides  more  participation  at  technical  working  group  level  when 
developing  the  standard. 

Human  resources  could  be  split  between  DoD  and  sponsored  agency 
participation  to  balance  objectivity  with  understanding  the 
requirements. 


Cons. 

CALS  less  in  control  of  influencing  implementable  content  of  STEP, 
since  other  test  beds  (e.g.,  European  and  Asian  activities)  would 
drive  the  standard's  content. 

Potentially  less  implementable  APs  as  output,  since  specification 
text  would  be  written  on  assumption  that  it  was  implementable. 
When  human  resources  move  on,  no  retention/record  of  corporate 
knowledge  occurs. 
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Less  credibility  when  introducing  specification  content  because  test 
data  does  not  exist;  therefore,  politics  rather  than  technical 
efficacy  may  win  out. 

Recommendation. A combination  of  alternative  1:  Develop  a Validation 
Testing  System  to  test  the  viability  of  the  STEP  draft  specifications  (e.g., 
application  protocols):  and  alternative  2:  If  a Validation  Testing  System 
to  test  the  viability  of  the  STEP  draft  specifications  is  developed,  NIST 
should  be  the  host  site.  Identifying  DoD  requirements  for  the  use  and 
functionality  of  STEP  is  necessary  early  in  the  process,  so  the  technical 
specifications  can  be  drafted  and  introduced  into  the  international  arena. 
DoD  benefits  by  supporting  a standards  testing  program  so  the  concepts 
and  ideas  are  evaluated  for  implementability  and  for  meeting  DoD 
requirements.  Installing  such  a program  of  standards  testing  within  NIST 
allows  leverage  of  the  technical  expertise  and  understanding  of  the 
international  standards-making  process. 

C.  Should  NIST  be  Responsible  for  Military  Specification 
Conformance  Testing? 

Under  the  CALS  initiative,  the  conformance  testing 
responsibilities  of  NIST  have  been  expanded  to  include 
conformance  testing  against  the  CALS  military  specifications. 

Prior  to  CALS  sponsorship  at  NIST,  conformance  testing  programs  were 
primarily  focussed  only  on  those  implementations  claiming  conformance 
to  the  FIPS  PUBS.  This  focus  was  based  on  two  reasons:  (1)  NIST  is 
responsible  for  the  development  of  FIPS  PUBS  and  (2)  FIPS  PUBS  are 
written  to  meet  the  requirements  of  all  federal  agencies.  With  the  advent 
of  CALS  military  specifications,  NIST  and  its  CEIO  sponsor  saw  a need  for 
finer  qualification  of  an  implementations's  claims.  These  specifications 
were  written  specifically  for  the  employ  of  one  federal  agency-the 
Department  of  Defense. 

Issue.  Although  NIST  develops  conformance  testing  services  for  those 
standards  which  are  FIPS  PUBS,  no  means  exist  to  perform  conformance 
testing  on  the  additional  requirements  imposed  on  the  supplier  through 
military  specifications. 
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Alternative  1.  Under  the  sponsorship  of  the  CEIO,  NIST  should  provide 
conformance  testing  services  for  the  CALS  military  specifications. 

Pros. 

NIST  has  conformance  testing  service  experience. 

Military  specification-level  conformance  testing  services  could  be 
harmonized  with  FIPS  conformance  testing  services. 

NIST  participates  in  the  international  efforts:  therefore,  can  best 
maintain  international  harmonization. 

NIST  is  an  objective  agent;  no  vested  interest  in  outcomes. 

Service  could  be  used  by  other  DoD  programs  with  similar  needs,  e.g., 
CIM;  therefore,  cost  sharing  could  occur. 


Cons. 

Military  specifications  are  application-specific,  and  tend  to  support 
only  one  federal  agency-DoD. 

DoD  expertise  would  not  be  built  if  NIST  provided  the  full  services. 
NIST  may  prefer  providing  conformance  testing  services  at  the  FIPS 
level  only,  since  FIPS  benefit  multiple  government  agencies. 

Alternative  2.  Under  the  CEIO  authority,  a DoD  site,  e.g,  the  Joint 
Interoperability  Test  Center  (JITC),  could  be  designated  to  provide 
conformance  testing  services  for  the  CALS  military  specifications. 

Pros. 

More  easily  controlled  by  CALS. 

Facility  could  be  used  by  other  DoD  programs  with  similar  needs, 
e.g.,  CIM;  therefore,  cost  sharing  could  occur. 

Potentially  one-stop-shopping  site  for  all  the  testing  needs: 
standards  testing,  conformance  testing,  acceptance  testing. 

Builds  DoD  expertise  in-house. 


Cons. 

May  not  be  attractive  to  other  federal  agencies  who  have  adopted  the 
CALS  military  specifications. 

Depending  on  commercial  testing  laboratory  interest,  could 
potentially  place  the  government  in  competition  with  the 
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commercial  sector  (other  testing  laboratories  performing  the  same 
testing). 

Most  likely  would  not  be  an  internationally-recognized  testing 
laboratory;  therefore,  requiring  suppliers  to  be  tested  multiple 
times  for  the  same  product.  Such  costs  would  ultimately  be 
reflected  in  DoD  procurements. 

All  costs  to  maintain  a DoD  testing  laboratory  are  DoD 
responsibility. 

Potentially  a conflict  of  interest,  e.g.,  testing  results  driven  by  the 
chain-of-command  desires. 

Since  CALS  military  specifications  are  the  application  level  of  FIPS 
implementations,  supporting  conformance  testing  programs  should 
not  be  developed  in  isolation. 

Recommendation. Alternative  1:  Under  the  sponsorship  of  the  CEIO,  NIST 
should  provide  conformance  testing  services  for  the  CALS  military 
specifications.  If  the  CEIO  desires  COTS  implementations,  a coordinated 
federal  and  international  approach  needs  to  be  regarded.  Given  NIST's 
current  mission,  standards  involvement  with  all  federal  agencies,  and  role 
in  the  international  conformance  testing  alliances,  the  CEIO  can  best 
leverage  this  activity  by  sponsoring  NIST  for  the  military  specifications 
conformance  testing  services. 

D.  Should  NVLAP  be  Used  for  Testing  Laboratory 

Accreditation? 

NVLAP  is  comprised  of  a series  of  laboratory  accreditation 

programs  which  are  established  on  the  basis  of  requests  and 
demonstrated  need.  The  use  of  formal  accreditation  raises  a lot  of 
issues  for  a sponsor  such  as  Department  of  Defense.  As  a big  buyer  of 

digital  data,  the  CEIO  wants  to  ensure  to  the  best  of  its  ability  that 

software  generating  the  digital  data  is  conforming.  This  allows  a better 
chance  for  successful  interoperability  testing,  less  cost  when  conducting 
acceptance  testing,  and  supports  industry  for  economic  competitiveness 
with  other  national  bodies.  All  of  these  issues  have  to  be  balanced 
against  the  price  of  initiating  and  maintaining  an  accreditation  program. 
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Issue.  Accreditation  of  testing  laboratories  is  often  viewed  as  an 
expensive  overhead  both  in  actual  dollars  required  to  initiate  and  time 
required  to  establish  such  a program.  Should  the  CEIO  invest? 

Alternative  1 . Provide  testing  laboratory  accreditation  for  first,  second, 
and  third  party  testing  laboratories  through  NVLAP.  The  upper  part  of 
Figure  5 shows  the  general  process  for  accreditation. 


Memorandum  Mutual 
of  Understanding  Recognition 


or 


or 


Figlire  5:  Testing  Laboratory  Acceptance 


Pros. 

Better  ensures  the  quality  and  accuracy  of  test  data. 

Provides  some  assurance  of  the  technical  proficiency  and 
competence  of  a laboratory  to  assess  an  implementation's 
conformance  to  a set  of  prescribed  standards  [4576-91]. 

Better  opportunity  for  international  harmonization. 

Ensures  consistent  test  methods  and  test  tools  are  being  employed 
during  the  evaluation  process. 
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balanced,  objective  evaluation  thus  contributing  toward  market 
confidence. 


Cons. 

More  labor-intensive  to  implement  than  the  alternatives  in  the 
lower  part  of  Figure  5. 

More  cost-intensive  to  implement  than  the  alternatives  in  the  lower 
part  of  Figure  5. 

Depending  on  supply  and  demand,  CALS  may  have  to  forever  subsidize 
those  conformance  testing  services  which  are  not  self-sustaining. 
May  require  more  time  to  establish  accreditation  program. 

Alternative  2.  Allow  the  alternatives  for  laboratory  recognition  that  are 
shown  in  the  lower  part  of  Figure  5.  Current  examples  of  such 
arrangements  are  the  conformance  testing  services  for  language 
compilers,  database  language  SQL,  and  the  graphics  standards  (CGM  and 
Graphical  Kernel  System  {GKS}). 

Pros. 

Establishes  mutually  recognized  testing  laboratories  among  those 
interested  in  providing  some  level  of  a consistent  conformance 
testing  program  for  a specific  standard. 

Usually  quicker  to  establish  arrangements. 


Cons. 

Requires  a Memorandum  of  Understanding  (MOU)  to  globally  ensure 
mutual  reciprocity  at  the  testing  laboratory  level. 

Since  testing  laboratory  acceptance  is  via  agreement  only,  cannot 
ensure  consistent  testing  procedures  across  testing  laboratories. 
Testing  laboratory  agreements  often  built  on  established 
reputations,  not  necessarily  on  supply  and  demand;  therefore, 
difficult  to  create  new  testing  laboratories  if  demand  warrants. 

May  be  suspect  to  supplier  or  enterprise  user. 

Alternative  3.  CEIO  could  permit  a supplier's  declaration  (supplier's 
declaration  replaces  use  of  "self-certification"). 
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Pros. 


No  up-front  cost  to  CALS. 

Marketing  claims  are  robust,  painting  successful  progress  of  CALS 
initiative. 


Cons. 

No  consistent  set  of  test  suite/tools  or  procedures  for  comparison. 
Not  performed  by  objective,  unbiased  party. 

Assurance  for  conformance  only  as  good  as  trust  in  supplier. 

Often  quality  assurance  or  acceptance  testing  (e.g.,  performance, 
robustness,  data  set  testing)  versus  conformance  testing  practices 
performed,  forcing  conformance  testing  analysis  on  each  buying 
enterprise  user  as  part  of  acceptance  testing. 

Practice  not  commonly  recognized  internationally  for  information 
technology  standards'  implementations. 

Alternative  4.  CEIO  could  recognize  specific  testing  tools  or  test  suite 
only,  making  them  commercially  available  through  National  Technical 
Information  Service  (NTIS). 

Pros. 

CEIO,  suppliers,  and  enterprise  users  would  not  be  faced  with 
additional  costs  associated  with  accreditation. 

Allows  cheaper  testing  for  repeated  in-house  use. 

Provides  something  for  use  to  the  supplier  quicker  than  waiting  for 
an  accreditation  program. 

Assists  supplier  during  implementation  development  and  component 
testing. 


Cons. 

Does  not  ensure  the  application  of  such  tools  would  be  consistent 
between  enterprise  users;  therefore,  could  not  ensure  consistency  of 
results. 

Results  would  not  be  recognized  internationally. 

Depending  on  selection  of  CEIO  testing  tools  or  test  suite,  results 
may  not  be  recognized  by  other  federal  agencies  using  the  military 
specifications. 


39 


Testing  tools,  test  suite,  and  test  methods  may  not  be  consistent 
with  FIPS  conformance  testing  programs. 

Recommendation. Alternative  1:  Provide  testing  laboratory  accreditation 
for  first,  second,  and  third  party  testing  laboratories  through  NVLAP.  The 
upper  part  of  Figure  5 shows  the  general  process  for  accreditation.  When 
suppliers  are  driven  either  by  policy  or  the  acquisition  process,  a market 
is  created  for  a conformance  testing  service.  Accreditation  should  be 
implemented  to  keep  the  process  objective,  maximize  interns^'np^' 
consensus,  and  minimize  the  CEIO  investment  by  creating  a self- 
sustaining  vehicle  to  perform  conformance  testing. 

E.  Should  the  CTN  Perform  Interoperability  Testing? 

As  an  overall  tasking,  the  CTN  Office  is  responsible  for 
government  user  application,  interoperability,  and  other  related 
testing.  Interoperability  testing  should  be  part  of  acceptance 
testing  since  the  enterprise  user  has  the  ultimate 
responsibility  for  interoperability.  When  a client  sends  in  data  files 
or  an  implementation  to  undergo  testing  via  the  CTN,  the  receiving  test 
bed  performs  both  standards  and  interoperability  testing  at  the  same 
time.  The  extent  of  interoperability  testing  is  only  limited  by  the 
hardware  and  software  configuration  options  available  at  that  test  bed. 

Issue.  The  CTN  performs  standards  testing  and  interoperability  testing 
without  routinely  performing  conformance  testing.  This  CTN  practice  is 
not  the  same  sequence  as  occurs  in  general  practice. 

Alternative  1.  Offer  CTN  services  on  two  separate  levels:  first  standards 
testing  against  one  system.  Then,  after  client  can  offer  proof  of 
conforming  implementation  (e.g.,  via  Validated  Products  List),  allow 
client's  return  to  the  CTN  for  interoperability  testing. 

Pros. 

Flows  with  the  serial  process  of  testing  that  normally  occurs  (as 
defined  in  Figures  1 and  2). 

Reinforces  to  industry  and  enterprise  users  that  conformance  is 
necessary  prior  to  interoperability  testing. 


40 


Better  prediction  of  time  required  to  test  each  implementation  in 
the  standards  testing  environment  against  the  CTN  test  suite. 
Reduces  time  required  to  process  a given  implementation. 

Allows  more  time  for  more  volunteer  supplier  participants. 


Cons. 

More  administrative  costs  since  a new  test  report  would  be 
generated  each  time  an  implementation  was  tested  in  a different 
environment. 

Preparatory  time  and  cost  would  have  to  be  repeated  each  time  the 
same  implementation  returned  for  more  testing. 

Alternative  2.  Offer  CTN  services  for  both  standards  testing  and 
interoperability  testing  at  once  (status  quo). 

Pros. 

Program  manager  gets  his  bottom  line:  interoperability. 

Reinforces  that  CALS  specifications  work  in  multiple  environments. 


Cons. 

Interoperability  performed  on  two,  not  necessarily  conforming, 
implementations  requires  more  time  since  no  common  ground  is 
established. 

Much  redundancy  in  testing  occurs  from  implementation  to 
implementation:  therefore,  more  expensive. 

CEIO  paying  for  what  the  individual  program  manager  should  pay  for. 
Interoperability  testing  subtracts  manpower  from  standards  testing. 
Sends  a message  to  program  manager  that  interoperability  is  not  a 
program  manager's  issue;  the  CEIO  will  always  concern  itself  with 
the  government  user's  interoperability  requirements. 

Enterprise  user  may  not  bother  with  conformance  testing  since 
interoperability  is  often  the  greatest  concern. 

Alternative  3.  Offer  CTN  services  for  only  standards  testing. 

Pros. 

More  cost  effective  for  CEIO. 

Less  system  maintenance  and  manpower  required. 
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More  time  to  test  draft  specifications  prior  to  publication. 

More  time  to  assist  in  conformance  testing  tool  and  test  suite 
development  and  assessment. 

Focusses  funding  and  resources  on  assessment  of  the  standard. 


Cons. 

Less  testing  of  implementations  in  multiple  environments. 
Alternative  4.  Offer  interoperability  testing  only  as  part  of  acceptance 
testing  for  a given  procurement. 

Pros. 

Would  be  funded  by  the  individual  DoD  program,  not  the  CEIO. 

Would  flow  with  the  serial  process  of  testing  that  normally  occurs 
(as  defined  in  Figure  2). 


Cons. 

Need  for  such  a service  would  not  be  consistent;  therefore,  someone 

must  fund  the  equipment  maintenance  and  staff  overhead. 

Recommendation.  A combination  of  both  alternative  3;  Offer  CTN  services 
for  only  standards  testing;  and  alternative  4:  Offer  interoperability 
testing  only  as  part  of  acceptance  testing  for  a given  procurement,  is 
appropriate.  To  better  control  the  funding  and  associated  activities,  the 
CEIO  should  support  the  CTN  for  standards  testing  only.  This  would  more 
easily  allow  identifiable  milestones  to  achieve  closure  for  any  given 
standards  testing  program. 

As  the  need  arises,  the  CTN  test  beds  should  also  be  able  to  support  their 
individual  agency  to  perform  various  aspects  of  acceptance  testing, 
including  interoperability.  This  takes  advantage  of  the  technical 
expertise  already  established  at  the  test  bed,  allows  particular  agency 
procurement  needs  to  be  met,  and  requires  the  individual  program 
managers  to  fund  the  effort.  Direct  CTN  involvement  with  their  agencies 
also  surfaces  real-world  issues,  often  necessitating  improvement  to  the 
military  specifications  themselves. 
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F.  Should  the  CTN  Acclaim  CALS  Compliance? 

To  affirm  CALS  compliance  only  for  DoD-owned  systems.  In 

addition  to  its  scope  for  standards  and  interoperability  testing,  the  CTN 
was  given  authority  to  affirm  CALS  compliance  for  those  implementations 
already  on  a DoD  system.  This  has  created  a perception  in  the  CALS 
community  that  any  client  participating  in  the  voluntary  CTN  program  is, 
in  effect,  undergoing  conformance  testing  and  can  make  such  claims. 

Issue.  Whether  the  CTN  should  affirm  CALS  compliance  only  for  DoD- 
owned  systems. 

Alternative  1 . Allow  CTN  test  beds  to  claim  CALS  compliance  for  those 
implementations  already  on  DoD  systems. 

PrfiS- 

Provides  complete  continuum  of  standards  testing,  conformance 
testing,  and  acceptance  testing  (interoperability  testing  aspects) 
for  the  DoD  user. 

Provides  in-house  DoD  user  service. 


Cons. 

Presents  an  "uneven  playing  field."  Only  those  suppliers  who  are 
already  on  DoD  systems  have  an  opportunity  for  such  evaluation. 

Such  an  evaluation  is  occurring  AFTER  the  product  is  purchased  and 
DoD  is  already  financially  committed. 

Creates  public  image  of  confusion  as  to  who  is  responsible  for 
conformance  testing  of  products. 

Negatively  impacts  competitive  procurement  for  those  requests  for 
proposals  which  specify  "must  be  CTN-approved." 

CEIO  continues  to  fund  (forever?)  service  requirements  for  testing. 
Measurement  of  conformance  by  CTN  may  not  be  consistent  with 
other  conformance  testing  programs  established  by  NIST. 

Does  not  allow  same  service  for  other  federal  agencies  using  the 
military  specifications. 

Such  complete  coverage  of  all  testing  activities  by  one  organization 
does  not  allow  for  a checks-and-balances  system. 
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Alternative  2.  Allow  CTN  to  offer  conformance  testing  service  for  CALS 
initiative  to  any  client. 

Pros. 

Provides  complete  continuum  of  standards  testing,  conformance 
testing,  and  acceptance  testing  (interoperability  testing  aspects) 
for  the  DoD  user. 

Provides  in-house  DoD  user  service. 

Alleviates  public  confusion  of  who  is  responsible  for  conformance 
testing. 


Cons. 

CEIO  continues  to  fund  (forever?)  service  requirements  for  testing. 
Measurement  of  conformance  by  CTN  may  not  be  consistent  with 
other  conformance  testing  programs  established  by  NIST. 

Does  not  allow  same  service  for  other  federal  agencies  using  the 
military  specifications. 

Such  complete  coverage  of  all  testing  activities  by  one  organization 
does  not  allow  for  a checks-and-balances  system. 

Could  not  predict  workload,  therefore  may  adversely  impact 
standards  testing. 

Does  not  lend  itself  to  supporting  international  harmonization  of 
conformance  testing  programs  for  standards. 

Could  pose  a conflict  of  interest  if  performing  a conformance 
testing  assessment  on  a candidate  supplier  under  a Request  for 
Proposal. 

Alternative  3.  Focus  CTN  activities  on  acceptance  testing. 

Pros. 

CEIO  would  not  have  to  fund  acceptance  testing  service:  individual 
program  managers  would. 

CTN  technical  expertise  already  established. 

Test  beds  have  readily  available  "on  demand"  resources. 

Could  delineate  end  of  CTN  standards  testing  from  special  service 
acceptance  testing  requests. 
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Cons. 


Could  not  predict  workload,  therefore  may  adversely  impact 
standards  testing. 

Recommendation.  Alternative  3:  Focus  CTN  activities  on  acceptance 
testing,  complements  the  previous  recommendation  for  separation  of 
standards  testing  from  interoperability  testing. 

The  following  issues  were  not  raised  in  the  previous  text,  but  are  also 
some  questions  facing  the  CALS  community. 

G.  Could  the  CEIO  Use  Supplier-Developed  Testing 
Suites/Tools? 

Software  and  testing  tools  developed  to  accomplish  component  testing  by 
the  supplier  may  be  an  asset  to  the  CALS  initiative  if  they  were  made 
available  to  accomplish  specific  testing  to  CALS  requirements.  Tools  for 
a commercial  conformance  testing  service  are  very  expensive  to  develop. 
Industry  also  faces  a corporate  overhead  cost. 

Corporate  development  of  proprietary  testing  tools  to  evaluate  the 
"goodness"  and  "conformity"  of  their  implementations  is  usually  done  in- 
house.  It  is  costly  for  the  corporation  to  invest  in  such  testing,  but  it  is 
deemed  worthwhile  for  such  testing  to  take  place.  Since  this  testing  is 
most  always  done  in  isolation  of  other  corporations,  the  cost  to  develop 
and  implement  such  testing  in  tool  development  and  test  bed  maintenance 
is  high.  Such  costs  are  ultimately  passed  onto  DoD  through  procurements 
of  the  implementation  itself. 

Issue.  Should  the  CEIO  try  to  leverage  industry's  test  suite  and  tool 
development  resources  to  benefit  conformance  testing  and  reduce 
industries'  investment  costs? 

Alternative  1.  Such  proprietary  testing  tools  could  be  made  available  to  a 
CALS  conformance  testing  program  for:  (1)  use  if  complete  or  (2) 
adaptation  if  incomplete. 


45 


Pros. 


May  diminish  the  cost  of  conformance  testing  software  development. 
May  reduce  the  individual  corporate  requirements  for  in-house 
testing,  thus  reducing  the  overhead  costs. 

The  corporation(s)  providing  such  software  may  recoup  a portion  of 
their  investment,  thus  providing  some  incentive  for  sharing. 
Potentially  quicker  response  to  provide  conformance  testing  service 
than  waiting  for  the  test  suite/test  tools  to  be  coded  from  scratch. 


Cons. 

Such  testing  tools  may  be  viewed  as  a marketing  advantage  for 
commercial  sale  of  the  implementation. 

Often  the  testing  tools  are  developed  on  a system  configuration  not 
easily  portable  for  other  system  use. 

It  may  not  be  possible  to  separate  the  conformance  tests  from  those 
used  to  meet  other  requirements  such  as  robustness,  performance, 
interoperability. 

Alternative  2.  An  industrial  consortium  could  be  established  to  fund  tool 
development,  motivating  corporations  to  "buy-in"  by  providing  reduced 
costs  for  access  to  a test  bed. 

Pros. 

Corporate  cost  to  participate  should  be  less  than  cost  to  internally 
develop  test  tools  and  test  bed. 

CALS  ElO  investments  can  be  combined  with  other  funds  for 
developing  a test  bed  to  perform  conformance  or  acceptance  testing. 
Such  a consortia  could  also  serve  as  a supplier's  forum  to  identify 
deficiencies  in  the  national/international  standards  and  military 
specifications. 

Cons. 

Industry  participation  may  not  be  substantial  enough  to  establish  a 
test  bed  in  a timely  fashion  convenient  to  the  consortium  members. 
Some  corporations  may  view  "sharing"  as  cutting  into  their 
competitive  edge. 

Test  bed  may  already  be  established  in  a corporation  and  therefore 
not  worth  the  participation. 


46 


Industry-driven/developed  test  suite/test  tools  may  be  biased 
toward  consortium  participants'  implementations. 

Recommendation. Alternative  1:  Such  proprietary  testing  tools  could  be 
made  available  to  a CALS  conformance  testing  program  for:  (1)  use  if 
complete  or  (2)  adaptation  if  incomplete.  Past  experience  has  shown 
little  cooperation  by  industry  to  provide  testing  tools  the  supplier  has 
already  developed  to  perform  its  internal  component  testing.  Such 
development  work  is  expensive,  and  most  suppliers  see  little  ! in 

sharing  their  internal  testing  tools.  There  have  been  a few  exceptions  to 
this,  however,  where  test  cases  or  tools  have  been  provided.  Although 
work  is  usually  required  to  modify  or  complete  such  donations  of  software 
for  conformance  testing,  they  have  served  as  a foundation. 

H.  Should  the  CEIO  Invest  in  the  Development  of 
Conformance  Testing  Services? 

In  order  to  establish  commercial  off-the-shelf  (COTS)  implementations  of 
CALS  military  specifications  and  standards,  conformance  testing  services 
have  to  be  available.  ("Conformance  testing  services"  in  this  context 
include  everything  necessary  to  establish  such  a service  for  a given 
standard:  the  development  or  assessment  of  a test  suite/test  tool, 

establishing  the  accreditation  criteria,  and  writing  the  policies  and 
procedures  for  the  conformance  testing  service.)  The  CEIO  faces 
continuing  budgetary  restrictions  which  affects  the  priorities.  NIST  may 
or  may  not  establish  conformance  testing  services  for  those  FIPS  on 
which  the  military  specifications  are  based.  Even  if  a FIPS  conformance 
testing  service  was  being  established,  it  may  not  meet  the  requirements 
for  CALS  specifications.  Often  there  are  preliminary  syntactic  and 
semantic  requirements  in  the  FIPS  which  the  military  specification 
assumes;  however,  the  military  specifications  impose  additional 
constraints  on  the  application  of  the  FIPS  in  a DoD  environment. 

Issue.  Should  the  CEIO  fund  the  development  of  a conformance  testing 
service  at  either  the  FIPS  or  associated  military  specification  level? 
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Alternative  1 . CEIO  should  fund  conformance  testing  service  development 
at  both  the  FIPS  and  associated  military  specification. 

Pros. 

Provides  consistency  across  FIPS  and  military  specification 
conformance  testing  programs. 

Keeps  DoD  in  sync  with  the  practices  that  other  government  agencies 
may  be  employing  for  conformance  testing  since  FIPS  apply  to  all 
federal  agencies. 

Ensures  CEIO  will  get  conformance  testing  services  to  meet  CALS 
requirements. 

Establishes  CALS  standards  into  commercial  off-the-shelf 
implementations. 

Would  help  advance  CALS  standards  into  other  federal  agencies,  at 
least  at  the  FIPS  level;  easier  for  communicating  across  agencies. 
May  be  most  cost-efficient  to  support  a FIPS  conformance  testing 
service  only,  due  to  the  tailored  application  disposition  of  the 
military  specification;  something  is  better  than  nothing. 


Cons. 

Most  expensive. 

Given  the  combination  of  other  priorities,  timeliness  is  sometimes 
impacted  because  of  multi-year  effort. 

Makes  CEIO  responsible  for  FIPS  conformance  testing  costs  when 
other  government  agencies  should  be  contributing. 

Alternative  2.  CEIO  should  fund  only  the  conformance  testing  service 
requirements  associated  with  the  military  specification. 

Pros. 

Cheaper  than  funding  FIPS  conformance  testing  service  also. 

Better  CEIO  control  over  testing  tools/service  since  only  applicable 
to  military  specification. 


Cons. 

Since  most  of  the  military  specifications  subsume  any  syntactic  and 
semantic  FIPS  requirements,  CEIO  would  be  reinventing  the  wheel 
for  a lot  of  the  work. 
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If  a conformance  testing  service  were  established  independently  of 
CALS,  coordination  with  a FIPS  conformance  testing  service  or  other 
international  activity  may  be  lacking. 

May  not  be  recognized  by  other  government  agencies,  therefore 
forcing  suppliers  to  undergo  multiple  assessments. 

Recommendation.  Alternative  1 : CEIO  should  fund  conformance  testing 
service  development  at  both  the  FIPS  and  associated  military 
specification.  Unless  some  other  federal  agency  has  identified  a strong 
enough  interest  in  the  FIPS  to  support  the  development  of  a FIPS 
conformance  testing  service,  such  a service  may  not  be  established.  By 
supporting  first  the  development  of  a conformance  testing  service  which 
covers  the  FIPS,  then  the  military  specification,  the  CEIO  can: 

Assess  whether  further  investment  in  the  military  specification 
level  is  appropriate  and  efficient. 

Stay  harmonized  with  industry  and  other  government  agencies  for 
those  FIPS  which  are  based  on  national  and  international  standards. 
Ensure  a conformance  testing  service  will  be  established  which  will 
meet  their  requirements. 

1.  How  Should  the  CEIO  Ensure  Conformance  Testing 
Becomes  Part  of  the  Way  of  Doing  Business? 

Even  if  the  CEIO  establishes  conformance  testing  services,  the 
requirement  has  to  be  instilled  into  both  the  Defense  community  and  the 
supporting  industry.  There  are  a lot  of  misunderstandings  with 
terminology,  what  a specific  testing  activity  provides,  and  the  advent  of 
receiving  digital  versus  hard  copy  data.  Both  the  enterprise  user  and  the 
supplier  may  not  appreciate  the  full  value-added  in  requiring  and 
participating  in  conformance  testing  activities. 

Even  though  CALS  is  funding  various  conformance  testing  activities  and 
services,  the  ultimate  measure  of  success  is:  whether  the  procuring 
enterprise  user  requires  conformance-tested  implementations  in 
contracts  and  whether  the  supplier  is  motivated  to  apply  for  conformance 
testing  without  a request  for  proposal  requirement.  The  CEIO  needs  to 
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consider  solutions  which  educate  and  support  both  the  enterprise  user  and 
the  supplier. 

Issue.  Even  if  conformance  testing  services  existed  for  all  the  CALS 
military  specifications,  there  is  no  guarantee  enterprise  users  will  take 
advantage  of  such  services. 

Alternative  1.  The  CEIO  should  request  changes  to  the  Federal  and  Defense 
Acquisition  Regulations  (FAR/DAR)  so  only  those  implementations  vyhich 
have  undergone  successful  conformance  testing  are  eligible  for  defense 
purchase. 

Pros. 

Raises  the  conformance  testing  requirement  above  the  CEIO 
initiative,  thus  incorporating  CEIO  requirements  into  the  defense  and 
federal  infrastructure. 

Suppliers  are  more  apt  to  be  motivated  to  undergo  conformance 
testing  prior  to  request  for  proposal  inducement. 

Fosters  commercial  off-the-shelf  availability  of  conforming 
implementations. 

Provides  universal  common  ground  of  implementation  evaluation, 
thus  reducing  the  guesswork  of  "buyer  beware." 

Provides  the  clout  of  "law"  for  government  users  to  mandate  such 
practices. 


Cons. 

The  process  for  change  is  tedious  and  slow,  due  to  impact  on  more 
than  just  DoD  and  CEIO. 

Inappropriately  scoped,  timed,  or  supported  regulations  can  have  a 
tarnishing  impact  on  conformance  testing  (and  the  standard  of 
interest). 

Alternative  2.  Develop  policy  (e.g.,  only  those  implementations  published 
in  the  Validated  Products  List  will  be  considered)  and  guidelines  (e.g.,  MIL- 
HDBK-59)  which  implement  conformance  testing  into  the  purchasing 
infrastructure. 
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Pros. 


Provides  primary  source  of  reference  for  enterprise  users. 

The  Validated  Products  List  has  international  recognition  as  a 
source  document  on  implementations  which  have  undergone 
conformance  testing. 

Government  user's  job  is  made  easier  when  writing  requests  for 
proposals  (RFPs). 

Creates  a more  consistent  way  of  doing  business. 

Relatively  easy  to  develop  policy  and  guidelines. 


Cons. 

Ineffective  unless  conformance  testing  readily  available. 

May  be  difficult  to  enforce  policy  and  guidelines. 

Since  the  Validated  Products  List  is  only  published  quarterly,  it  still 
requires  the  prompt  issuing  of  a hard  copy  certificate  for  a 
conforming  implementation  which  has  successfully  undergone 
conformance  testing. 

Adoption  of  Validated  Products  List  requires  CEIO  comply  with  NIST 
acceptance  of  test  suite/tools,  testing  laboratory  accreditation 
procedures,  and  means  of  reporting  test  results. 

Alternative  3.  CEIO  provide  funding  to  establish  and  ensure  conformance 
testing  services  exist  for  each  of  the  CALS  specifications. 

Pros. 

Ensures  policies  and  guidelines  are  meaningful. 

Ensures  the  CEIO  that  testing  laboratories  will  always  exist  for 
conformance  testing  of  their  military  specifications. 

"Tailors"  conformance  testing  services  to  meet  CEIO  requirements. 
(In  the  case  of  SGML,  other  industries  [i.e..  Aerospace  Industry 
Association]  have  their  own  requirements  for  ensuring  conformance 
testing  tools  exist  specific  to  their  industrial  application. 

CEIO  can  control  their  budgetary  priorities  and  focus  on  some 
standards  over  others  to  meet  flagship  program  requirements. 


Cons. 

More  expensive. 

Face  timing  constraints  given  a limited  budget. 
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The  CEIO  carries  the  burden  by  itself. 


Alternative  4:  Leverage  existing  commercial  efforts  and  any  commercial 
demands  for  conformance  testing. 

Pros. 

Takes  advantage  of  what  is  already  a resource. 

Keeps  DoD  aligned  with  industry  thinking. 

Supports  COTS  implementations  along  with  industry. 

Less  expensive. 


Cons. 

Focus  often  too  specialized  for  particular  industry. 

Often  industry  demand  for  conformance  testing  based  on  government 
demands;  therefore,  government  usually  in  the  lead. 

Recommendation  .All  four  alternatives  should  be  employed: 

Alternative  1:  The  CEIO  should  request  changes  to  the  Federal  and 
Defense  Acquisition  Regulations  (FAR/DAR)  so  only  those 
implementations  which  have  undergone  conformance  testing  are 
eligible  for  Defense  purchase.  If  this  alternative  is  successfully 
accomplished,  it  would  force  the  hand  for  alternative  2;  however,  it 
is  very  difficult  to  affect  such  regulations.  Therefore,  the  CEIO  will 
want  to  employ  other  means  to  ensure  conformance  testing  becomes 
a way  of  doing  business. 

Alternative  2:  Develop  policy  (e.g.,  only  those  implementations 
published  in  the  Validated  Products  List  will  be  considered)  and 
guidelines  (e.g.,  MIL-HDBK-59)  which  implement  conformance  testing 
into  the  purchasing  infrastructure. 

Alternative  3:  CEIO  provide  funding  to  establish  and  ensure 
conformance  testing  services  exist  for  each  of  the  CALS 
specifications.  If  the  CEIO  wants  to  ensure  DoD  can  buy  conforming 
COTS  CALS  implementations,  conformance  testing  needs  to  be  built 
into  the  CEIO  budget.  Historically,  conformance  testing  activity  has 
been  driven  randomly  by  the  priorities  of  the  moment. 
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Alternative  4:  Leverage  existing  commercial  efforts  and  any 

commercial  demands  for  conformance  testing.  The  automobile 
industry  both  here  in  the  United  States  and  abroad,  has  recognized 
the  benefits  in  using  standards.  As  an  example,  IGES,  a U.S.  national 
standard,  already  has  supporting  conformance  testing  services 
offered  in  the  United  Kingdom  for  any  application,  as  well  as  in 
Germany  for  the  automotive  industry.  Electronic  publishing  and  the 
use  of  SGML  have  also  brought  conformance  testing  requirements  to 
the  forefront  in  Europe. 

In  order  to  "ensure"  conformance  testing  becomes  part  of  the  CALS 
infrastructure,  the  CEIO  should  invest  time  and  resources  into  all  of  the 
alternatives;  any  one  alternative  will  not  give  the  CALS  initiative  full 
benefit. 
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VI.  CONFORMANCE  TESTING  STATUS  OF  CALS 
STANDARDS . 

The  following  provides  a status  of  conformance  testing  programs  of 
standards  already  adopted  under  the  CALS  initiative,  as  well  as  those 
which  are  related  and  may  be  necessary  to  support  CALS  applications. 
Where  appropriate,  a distinction  is  made  on  the  status  of  conformance 
testing  programs  at  the  military  specification,  federal,  national,  or 
international  levels.  For  selected  standards,  a detailed  description  of  the 
supporting  test  suite,  test  tools,  or  conformance  testing  process  is 
provided. 

A.  CALS  Military  Standards  and  Specifications. 

1.  Automated  Interchange  of  Technical  Information  (MILSTD-I840). 

The  CALS  Evaluation  and  Integration  Office  (CEIO)  is  examining  the 
requirements  for  a conformance  testing  service  for  MILSTD-1840. 

2.  Initial  Graphics  Exchange  Specification  (IGES). 

The  only  commercial  conformance  testing  service  offered  for  IGES  is 
located  at  the  CAD-CAM  Data  Exchange  Technical  Centre  (CADDETC)  in  the 
United  Kingdom.  CADDETC  has  been  formally  accredited  by  their  national 
accreditation  body  to  perform  conformance  testing  for  IGES.  In  addition, 
there  are  several  U.S.  and  international  suppliers  that  offer  proprietary 
software  tools  to  test  IGES  data  files  of  implementations.  There  is  also  a 
public  domain  library  of  executable  test  suites  developed  by  the 
IGES/PDES  Organization  Test  Case  Design  Committee.  None  of  these  tools 
has  been  independently  assessed  for  its  capabilities  and  completeness. 

For  the  proposed  IGES  FIPS  (expected  publication  by  end  of  1992),  there  is 
no  commercially  available  service  available.  NIST  hopes  to  evaluate  those 
executable  software  tools  and  test  suites  available  as  a preliminary  step 
to  establish  a conformance  testing  service  for  IGES  FIPS. 

There  is  no  commercial  conformance  testing  service  available  for  the 
Digital  Representation  for  Communication  of  Product  Data  (Mil-D-28000). 
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There  are  several  suppliers  in  the  United  States  and  abroad  that  claim  to 
have  proprietary  testing  tools  to  evaluate  Mil-D-28000  data  files; 
however,  no  formal  assessment  of  the  quality  of  these  tools  has  been 
performed. 

3.  Standard  Generalized  Markup  Language  (SGML) . 

There  is  no  commercial  conformance  testing  service  to  test  the  SGML 
International  Standard  8879-1986.  However,  some  executable  test  suites 
exist,  but  none  of  these  test  suites/test  tools  has  been  independently 
assessed  for  quality  and  completeness.  A collaborative  effort  is 
underway  to  merge  a North  American  proprietary  executable  test  suite 
candidate  with  a proprietary  European  Community  (EC)  executable  test 
suite  candidate.  Both  developers  have  turned  over  their  copyright  to  the 
EC  Consortium  created  to  harmonize  the  international  testing  activities. 
This  would  enhance  the  probability  that  one  internationally-accepted 
executable  test  suite  could  be  used  for  conformance  testing  of  an  SGML 
implementation.  The  Graphic  Communications  Association  (GCA)  is 
assisting  in  the  harmonization  of  the  test  suites. 

In  1987,  under  the  sponsorship  of  the  CALS  Office,  NIST  developed  a 
publicly  available  validation  parser. 8 This  tool  was  built  to  support 
conformance  testing  of  both  IS  8879-1986  and  FIPS  PUB  152,  which 
adopts  the  ISO  standard.  Although  it  is  the  eventual  goal  of  the 
international  effort  to  harmonize  an  executable  test  suite  for  Manuals. 
Technical:  Markup  Requirements  and  Generic  Style  Specification  for 

Electronic  Printed  Output  and  Exchange  (MIL-M-28001 ),  no  such 
development  is  underway.  Under  CALS  sponsorship,  NIST  is  conducting  an 
independent  assessment  of  all  test  tools  available  for  conformance 
testing  FIPS  PUB  152,  as  well  as  MIL-M-28001. 

4.  Raster  Graphics . 

There  is  no  conformance  testing  service  available  for  Raster  Graphics 
Representation  in  Binary  Format.  Requirements  for  (MIL-R-28002)  or 


8’The  SGML  Parser,"  NTIS  publication  #PB87-2351 15/WCC. 

Available  through  the  National  Technical  Information  Service,  Springfield, 
VA  22161. 
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FIPS150:  however,  there  are  a few  proprietary  and  public  domain 
executable  test  suites  which  have  undergone  a technical  assessment  for 
quality  and  completeness.  Initial  focus  of  this  assessment  has  been 
Raster  Type  I.  Procedures  for  running  a conformance  testing  service  have 
been  beta-tested  under  the  sponsorship  of  the  CEIO  and  are  published  as 
NISTIR  4848,  Raster  Graphics  Validation.  Upon  completing  enhancements 
to  the  selected  conformance  testing  suite,  a conformance  testing  service 
for  Type  I is  expected  to  be  ready  at  NIST  by  the  end  of  1992.  Results  will 
be  published  in  the  Validated  Products  List.  An  accreditation  program  is 
currently  under  consideration  to  transfer  testing  laboratory 
responsibilities  from  NIST  to  accredited  first,  second,  and  third  party 
testing  laboratories.  NIST  is  also  in  the  process  of  evaluating  Type  II 
conformance  testing  tools. 

Figure  6 shows  the  raster  conformance  testing  process  steps  of  a client's 
Implementation  Under  Test  (lUT)  as  follows: 

Client  submits  the  Raster  Graphics  Request  for  Validation  form  to 
CSL.  The  client  shall  specify  the  testing  environment  and  the  type 
of  encoding  (Type  I or  II)  to  be  tested. 

NIST  instructs  the  testing  laboratory  to  assemble  and  send  to  the 
client  a conformance  test  package  consisting  of  instructions,  forms, 
and  two  sets  of  test  cases  formatted  in  accordance  with  MILSTD- 
1840  and  MIL-R-28002.  The  first  set  (or  document)  will  consist  of 
uncompressed,  binary  bitmap  images  of  various  sizes  and  contents. 
The  second  set  will  consist  of  images  of  various  sizes  and  similar 
(though  not  identical)  contents  to  the  first  set  but  will  be  encoded 
following  FIPS  PUB  150  (Type  I and  II)  and  FIPS  "Raster  DAP"  (Type  II 
only). 

Using  the  instructions  and  forms  received  from  the  testing 
laboratory,  the  client  processes  the  test  suite  through  the  lUT.  The 
set  of  bitmap  images  (set  1)  is  processed  creating  the  client's 
version  of  the  encoded  files,  and  the  set  of  laboratory  encoded  files 
(set  2)  is  decoded/decompressed  creating  the  client’s  version  of  the 
bitmap  files.  Both  sets  of  these  client  processed  images  will  then 
be  formatted  by  following  MILSTD-1840  and  MIL-R-28002.  At  the 
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option  of  NIST,  an  observer  may  be  assigned  to  watch  the  client's 
processing  of  the  test  suite. 


Figure  6:  Raster  Graphics  Conformance  Testing  Procedi^2i 
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The  client  sends  the  processed  files  to  the  testing  laboratory  for 
evaluation. 

The  testing  laboratory  will  evaluate  each  of  the  returned  set  of  files 
comparing  them  with  the  expected  results  to  determine  if  the 
client's  lUT  produced  the  expected  results.  The  client's  encoded 
images  from  set  1 will  be  compared  to  the  laboratory's  encoded 
versions  of  the  original  binary  bitmap  images.  The  client's  bitmap 
images  from  set  2 will  be  compared  to  the  laboratory's  bitmap 
images.  This  procedure  verifies  the  correctness  of  the  encoding  and 
compression/decompression  algorithm,  as  well  as  the  information 
contained  in  the  raster  data  file  header  records  regarding  the  image 
orientation,  dimension,  and  pel  density.  The  results  of  this 
comparison  will  be  evaluated  and  documented  for  inclusion  in  the 
Validated  Summary  Report.  The  evaluation  is  based  upon  only  the 
stated  system  configuration  and  does  not  indicate  what  the  system 
would  produce  under  a different  configuration. 

The  testing  laboratory  prepares  a Validated  Summary  Report 
containing  the  results  of  the  conformance  testing  [FS92]. 

5.  Computer  Graphics  Metafile  (CGM) . 

A full  conformance  testing  service  is  available  at  NIST  to  test  an 
implementation's  data  files  for  conformance  to  the  ISO  CGM  standard 
(which  is  adopted  through  FIPS  128),  as  well  as  the  Digital  Representation 
for  Communication  of  Illustration  Data:  CGM  Application  Profile  (MIL-D- 
28003).  The  results  of  such  testing  are  published  in  the  Validated 
Products  List.  To  date  (August  24,  1992),  one  conformance  assessment 
has  been  completed,  and  two  have  applied  for  the  service. 

NIST  anticipates  offering  a conformance  testing  program  for  CGM 
generators  by  the  end  of  1992.  Generator  testing  will  be  comprised  of  the 
metafile  conformance  testing  already  offered  plus  additional  steps. 

Figure  7 shows  the  CGM  conformance  testing  process  for  both  a client's 
metafile  and  generator. 
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Figure  7:  CGM  CONFORMANCE  TESTING  PROCESS 

Beginning  at  the  metafile  conformance  testing  level,  how  the  client 
generates  the  metafiles  for  conformance  assessment  is  invisible  to  the 
testing  laboratory;  the  conformance  testing  process  begins  with  the 
receipt  of  the  metafiles.  These  metafiles  are  syntactically  checked 
against  the  documented  requirements  of  the  international  CGM  standard 
and  the  CALS  application  profile.  The  results  are  published  in  a test 
report  and  published  (with  the  client's  consent)  in  the  Validated  Products 
List. 


Since  evaluation  of  a CGM  generator  is  passing  judgement,  not  just  on  the 
metafiles  able  to  be  output  but  on  the  CGM  implementation  itself,  it  is 
necessary  for  the  client  to  complete  a questionnaire.  This  questionnaire 
allows  the  client  to  identify  functional  capabilities  of  the 
implementation.  A test  suite  is  tailored  to  meet  the  specific  capabilities 
of  the  client's  implementation.  The  client's  CGM  generator  (which 
sometimes  includes  the  application  software,  other  times  calls  upon  it) 
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generates  CGMs  from  the  test  script  and  test  images.  These  CGMs  are 
analyzed  syntactically  and  semantically  for  completeness.  The  client  is 
responsible  to  provide  not  only  the  metafiles,  but  a hard  copy  graphical 
representation  of  the  metafile  (if  appropriate),  and  the  client's  internal 
format  used  to  generate  the  metafile. 

The  same  automated  syntactical  check  occurs  against  the  metafiles,  as 
well  as  automated  semantical  visual  and  completeness  checks.  At  the 
culmination  of  generator  conformance  assessment,  a test  report  i^ 
prepared,  and  again,  with  the  client's  permission,  the  results  are 
published  in  the  Validated  Products  List. 

6.  Contractor  Integrated  Technical  Information  Service  (MILSTD-CITIS) . 

Since  CITIS  is  not  yet  an  accepted  military  standard,  no  conformance 
testing  service  has  been  implemented. 

7.  Government  Open  Systems  Interconnection  Profile  (GOSIP) . 

NIST,  in  conjunction  with  the  Joint  Interoperability  Test  Center  (JITC)  at 
Fort  Huachuca,  Arizona,  offers  a U.S.  GOSIP  Register  Database  (GRD).  This 
GRD  provides  the  following  up-to-date  reference  information  relative  to 
FIPS  146  conformance  testing:  U.S.  GOSIP  abstract  test  suites;  assessed 
means  of  testing;  NVLAP-accredited  first,  second,  and  third  party  testing 
laboratories;  and  conformance  tested  GOSIP  products.  Information  about 
access  to  this  GRD  is  published  in  the  Validated  Products  List. 

Figure  8 is  the  conformance  assessment  process  overview  for  GOSIP  lUTs. 
The  conformance  assessment  process  involves  three  phases:  preparation 
for  testing:  test  operations:  and  test  report  production. 

The  preparation  for  testing  includes  producing  the  system  conformance 
statement.  Protocol  Implementation  Conformance  Statement  (PICS),  and 
Protocol  Implementation  Extra  information  for  Testing  (PIXIT);  choosing 
the  appropriate  abstract  test  method  and  abstract  test  suite  based  on  the 
PICS  and  PIXIT;  and  preparing  the  SUT  and  means  of  testing. 

The  test  operations  phase  involves  a static  conformance  review, 
conducted  by  analyzing  the  PICS  with  respect  to  the  relevant  static 
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conformance  requirements;  test  selection  and  parameterization  based  on 
the  PICS  and  PIXIT;  and  one  or  more  test  campaigns.  A test  campaign  is 
the  process  of  executing  the  parameterized  executable  test  suite  which  as 
been  produced  as  a result  of  the  test  selection  and  parameterization 
steps.  A test  campaign  consists  of  the  use  of  a configuration  of 
equipment  which  allows  protocol  exchanges  to  take  place  between  the  SUT 
and  the  test  system. 
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Figure  8:  GOSIP  Conformance  Assessment 


The  test  operations  phase  culminates  in  analyzing  the  results  and 
producing  a test  report. 
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8.  Very  High  Speed  Integrated  Circuit  (VHSIC)  Hardware  Definition 
Language  (VHDL). 

There  is  no  conformance  testing  service  available.  Virginia  Polytechnic 
Institute  (VPI)  and  State  University,  Blacksburg,  Virginia,  has  a 
proprietary  executable  test  suite  developed.  Seven  VHDL  CAD  suppliers 
provided  funding  for  the  initiative.  Vantage,  one  of  the  participating  CAD 
suppliers,  also  provided  an  initial  test  suite  that  was  jointly  developed  by 
Vantage  and  Intermetrics  under  contract  to  the  Air  Force  (AF  Wright 
Laboratory,  Solid  State  Electronics  Directorate).  The 
Vantage/Intermetrics  test  suite  had  high  granularity  tests.  VPI 
designated  test  points  at  the  paragraph  level,  then  associated  the  test 
cases  from  the  Air  Force  initiative,  affecting  the  initial  structure  of  the 
Vantage/Intermetrics  test  suite. 

VPI's  test  suite  software  is  free  to  universities  and  $2000  for  non- 
universities. The  VPI  test  suite  tests  primarily  (85%)  static  semantics, 
and  the  remainder  dynamic  semantics. 

The  Air  Force  has  assessed  the  VPI  test  suite  and  finds  it  the  most 
complete  in  existence  to  date.  The  Air  Force  would  like  to  find  funding  to 
build  upon  this  test  suite  to  make  it  more  complete.  There  has  also  been 
preliminary  discussion  with  the  CEIO  about  taking  over  the  VHDL  standard 
and  associated  conformance  testing  service  requirements  [JA92],  [JH92]. 

9.  Electronic  Design  Interchange  Format  (EDIF) . 

There  is  currently  no  formal  activity  in  conformance  testing  of  EDIF.  The 
University  of  Manchester,  United  Kingdom,  does  have  a reference 
implementation  which  has  not  undergone  an  independent  assessment; 
however,  if  a supplier  so  chooses,  the  supplier  may  send  EDIF  files  to  the 
University  of  Manchester  to  be  run  against  this  reference  implementation 
[RP92]. 

10.  Institute  for  Packaging  of  Electronic  Components  (IPC)  Standards . 

There  are  several  standards  by  the  IPC  350  series.  These  include: 

Printed  Board  Description  in  Digital  Form  (IPC-D-350). 
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Printed  Board  Drawings  in  Digital  Form  (ANSl/lPC-D-351). 

Electronic  Design  Data  Description  for  Printed  Boards  in  Digital 

Form  (ANSI/IPC-D-352). 

Automatic  Test  Information  Description  in  Digital  Form  (IPC-D- 

353) . 

Library  Format  Description  for  Printed  Boards  in  Digital  Form  (IPC-D- 

354) . 

Printed  Board  Automated  Assembly  Description  in  Digital  Form  (IPC- 

D-355). 

Printed  Board  Electrical  Test  Description  in  Digital  Form  (IPC-D- 

356). 

Guide  for  Digital  Descriptions  of  Printed  Board  and  Phototool  Usage 

per  IPC-D-350  (IPC-D-358). 

IPC  has  a commercially  available  certification  program  for  the  IPC-D- 
350,  revision  C.  The  testing  tools  were  developed  under  a collaborative 
arrangement  between  U.S.  IPC  and  Comargus  Data  Systems  in  Berlin, 
Germany.  The  software  is  written  to  cover  both  syntax  and  a graphical 
view  check.  Although  there  is  currently  no  formal  international 
recognition  of  IPC's  certification  program,  D-350,  revision  D is  about 
ready  to  be  adopted  as  an  international  lEC  standard  (lEC  1182-1).  The 
Japanese  have  shown  a strong  interest  in  D-350  and  have  driven  the 
enhancements  gained  by  lEC  1182-1.  The  international  adoption  of  D-350 
through  lEC  may  provide  a positive  incentive  for  Euro-Asian  adoption  of  an 
upgraded  version  (capturing  the  changes  imposed  by  revision  D)  of  the  U.S. 
IPC/German  test  tools.  The  tools  are  commercially  available  through  IPC 
and  Comargus  Data  Systems:  IPC  offers  a discount  to  IPC  members. 

The  IPC  conformance  testing  process  is  a collaborative  effort  among  the 
supplier,  an  offsite  testing  laboratory,  and  IPC.  Upon  request,  a supplier 
receives  and  reads  in  a test  vehicle  of  D-350  (revision  C).  Also  sent  are 
changes  the  supplier  is  responsible  to  make  to  his  file.  (These  changes 
are  tailored  to  each  conformance  assessment.)  The  supplier  makes  the 
designated  changes  to  his  file,  and  provides  a new  data  file  (an  unbundled 
D-350  set)  which  reflects  those  changes.  This  data  file  is  sent  to  the  IPC 
off-site  laboratory  where  the  file  is  run  through  a syntax  checker  that 
evaluates  files  for  correct  syntax;  "D-350  View"  software  which  verifies 
that  ail  features  of  the  test  vehicle,  including  any  required  edits,  are 
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present  and  correctly  defined  in  the  output  files;  and  a Manual  view  which 
verifies  presence  and  accuracy  of  features  in  photoplots  and  paper  check 
plots  [IPC91].  Assessment  of  the  required  changes  is  made,  and  the 
results  reported  to  IPC.  IPC  issues  a certificate  of  conformity  for  those 
implementations  meeting  the  requirements. 

Although  no  work  is  underway  to  provide  conformance  testing  services  for 
the  remainder  of  the  D-350  series,  I EC  is  expected  to  embrace  the  other  D- 
350  standards  as  part  of  the  lEC  1182  series.  This  adoption  is  expec+rr*  tn 
promote  further  requests  and  incentive  to  develop  conformance  testing 
services  for  the  remainder  of  the  D-350  standards  [BD92]. 

11.  Electronic  Data  Interchange  (EDI)  Transaction  Set  841 . 

The  EDI  Transaction  Set  841  is  called  out  as  an  alternative  transmission 
in  MILSTD-1 840B.  As  mentioned  previously,  the  overall  conformance 
testing  requirements  for  MILSTD-1840  are  currently  being  assessed. 
However,  since  the  national  EDI  standard  is  still  under  the  status  of  "draft 
standard  for  trial  use,"  no  known  conformance  testing  activity  or  test 
suite  development  is  occurring. 

B.  Future  Candidate  CALS  Standards. 

1.  Standard  for  the  Exchange  of  Product  Model  Data  (STEP). 

STEP  has  not  been  adopted  as  an  international  standard  to  date;  therefore, 
no  conformance  testing  service  is  available.  Development  of  abstract  test 
suites  to  support  STEP  application  protocols  is  underway,  and  initial  work 
on  a conformance  testing  system  is  being  funded  by  the  Navy 
Manufacturing  Technology  (MANTECH)  Program.  NIST  is  providing  technical 
and  managerial  oversight;  the  Industrial  Technical  Institute  (ITI)  of 
Michigan  is  doing  the  major  portion  of  the  technical  development.  A 
memorandum  of  understanding  has  been  established  between  the  National 
PDES  Testbed,  ITI,  and  PDES,  Inc.,  to  collaborate  on  the  development  of  an 
ATS.  Current  activities  in  support  of  STEP  include  proposing  the  abstract 
test  notation  for  ATSs  and  performing  a requirements  and  capabilities 
survey.  A survey  of  existing  STEP  tools,  which  may  contribute  toward  a 
conformance  testing  system,  has  been  completed.  An  additional 
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memorandum  of  understanding  has  been  drafted  between  the  National  PDES 
Testbed  and  the  European  Community's  (EC)  Conformance  Testing  Service 
(CTS)  initiative  through  a representative  of  the  CAD/CAM  EC  CTS  (Deputy 
Director  of  CADDETC)  to  harmonize  international  efforts  on  STEP 
conformance  testing. 
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Figure  9:  STEP  Conformance  Testing  System  Architecture 
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An  issue  already  identified  within  the  STEP  community  when  one 
discusses  conformance  testing  is  the  various  implementation  forms  (and 
associated  test  methods)  anticipated.  Initially,  STEP  technology  will  be 
based  on  file  exchange.  Figure  9 is  an  example  of  a conformance  testing 
system  architecture  for  the  file  exchange  implementation  form  using  a 
client's  preprocessor.  Eventually  the  goal  of  the  STEP  community  is 
product  data  sharing  through  shared  databases.  How  one  tests  the 

conformance  at  the  database  level  has  yet  to  be  determined. 
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2.  Database  Language  SQL. 


NIST  offers  a conformance  testing  service  for  implementations  which 
claim  conformance  to  Database  Language  SQL  FIPS  127-1  Level  II 
(including  the  integrity  enhancement  feature).  The  results  are  published 
in  the  Validated  Products  List. 


Figure  10  shows  a system  flow  diagram  for  basic  SQL  testing.  The  system 
flow  for  testing  the  integrity  enhancement  is  identical  in  structure. 
Running  an  SQL  test  suite  consists  of  five  steps. 


In  step  1,  the  schema  files  are  processed  in  some  implementor-defined 
manner,  perhaps  interactively. 


In  step  2,  a program  is  run  to  insert  values  into  six  of  the  base  tables. 
The  contents  of  these  six  tables  will  remain  unchanged  throughout 
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testing;  i.e.,  these  values  will  be  restored  by  each  program  which  changes 
them. 

In  step  3,  the  test  programs  are  run  to  interact  with  the  database  tables. 
Each  program  contains  logic  to  evaluate  the  database  responses  and 
determine  whether  a test  passes  or  fails.  This  pass/fail  decision  is 
recorded  by  inserting  a row  into  the  table  TESTREPORT.  In  general, 
programs  may  be  run  and  rerun  in  any  order. 

In  step  4,  values  are  inserted  into  the  reference  table  TESTINDEX.  This 
table  is  required  to  produce  the  automated  summary  report.  TESTINDEX  is 
also  a valuable  resource  to  testers,  since  it  can  be  queried  interactively 
to  create  a variety  of  useful  cross-references. 

In  step  5,  the  report  programs  are  run  to  produce  various  listings.  These 
listings  include:  required  tests  which  failed  (incorrect  performance): 
required  tests  which  failed  (missing):  tests  requiring  printouts  to 
demonstrate  FIPS  flagging:  optional  FIPS  sizing  tests;  and  a one-page 
summary  of  pass/fail  counts  by  category. 

To  test  another  programming  language  or  to  test  Interactive  SQL,  the 
tester  repeats  steps  3 and  5 with  another  test  suite  type. 

To  test  for  conformance  to  the  integrity  enhancement,  the  tester  repeats 
steps  1 through  5 with  additional  schema  files  and  another  set  of 
programs  [SQL92]. 

Total  NIST  SQL  Test  Suite  licensing  fee  for:  1 CPU  - $4,995,  multiple  CPUs 
(site-wide  license)  - $7,995;  1 CPU  (if  already  have  an  existing  NIST  SQL 
license)  - $3,995,  multiple  CPUs  (site-wide  license)  (if  already  have  an 
existing  NIST  SQL  license  - $6,495  [TS3.0]. 

3.  Information  Resource  Dictionary  System  (IRDS). 

NIST  has  developed  a draft  executable  test  suite  for  conformance  testing 
implementations  against  the  IRDS  FIPS  156  Command  Language,  and  is  in 
the  process  of  developing  supportive  test  tools.  Upon  completion,  NIST 
expects  to  offer  a conformance  testing  service  by  the  end  of  calendar  year 
1992. 
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A high-level  view  of  the  IRDS  conformance  testing  system  architecture  is 
provided  in  Figure  11.  The  following  are  the  components  of  the 
architecture: 


Figure  11:  IRDS  System  Architecture  (al9ii 


Command  Language  Test  Suite.  This  is  implemented  as  a set  of  text  files 
corresponding  to  the  test  sets  in  the  suite  and  is  supplied  by  NIST.  The 
format  of  each  file  is  precisely  that  of  a batch  file  of  commands. 

Command  Language  Interface  Driver.  This  software  component  processes 
the  Command  Language  Test  Suite  using  the  candidate  IRDS.  The 
organization  of  the  test  suite  assumes  that,  as  a practical  matter,  this 
processing  is  done  on  a test  set  by  test  set  basis.  For  each  command  in 
the  test  set,  the  Driver  echoes  and  executes  that  command;  formats  all 
direct  output,  including  error  and  success  messages  and  Output  IRD  and 
Output  IRD  Schema  displays;  and  writes  the  results,  in  the  required 
canonical  format,  to  the  Command  Language  Test  Results  File. 
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Command  Language  Test  Results  File  fTRF').  Produced  by  the  Command 
Language  Interface  Driver,  there  is  one  file  for  each  test  set.  This  file 
contains  a record  of  the  commands  that  were  run,  recording  for  each 
command  a copy  of  the  command  string  and  the  output  results  of  that 
command.  These  results,  written  in  the  NIST  specified  canonical  format, 
include  IRD  or  IRD  Schema  data,  success  messages,  and  return  codes. 

Command  Language  Expected  Results  File  (ERFT  This  file,  produced  by 
NIST,  contains  the  expected  results  of  the  processing  cf  the  Command 
Language  Test  Suite.  The  expected  results  are  stored  in  the  same  format 
as  in  the  TRF  to  allow  for  automated  comparison  of  the  test  results  with 
the  expected  results.  With  one  exception,  this  file  is  precisely  what  the 
TRF  would  look  like  if  the  candidate  IRDS  were  perfectly  conformant.  The 
exception  is,  for  Export  IRD  command,  the  ERF  contains  success  messages 
that  reference  pre-defined,  "expected"  export/import  files,  in  place  of 
TRF's  success  messages  that  contain  analogous  references  to  generated 
export/import  files. 

Command  Language  Conformance  Analyzer.  This  software  component, 
developed  by  NIST,  compares  the  test  results  in  the  TRF  (including  the 
contents  of  any  pointed-to  export/import  files)  with  the  expected  results 
in  the  ERF  (including  the  contents  of  any  pointed-to  export/import  files). 
The  Command  Language  Conformance  Analyzer  produces  a report  detailing 
all  tests  that  executed  successfully  and  all  tests  that  executed 
unsuccessfully,  along  with  any  discrepancies  or  errors  found  [AG91]. 

4.  Remote  Database  Access  (RDA). 

RDA  has  recently  become  an  international  standard.  No  decision  for  a 
conformance  testing  service  has  been  made;  however,  RDA  conformance 
testing  may  either  become  part  of  GOSIP  or  Database  Language  SQL's 
conformance  testing  service. 
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C.  Candidate  CALS  Standards  Contained  in  CIM  Technical 
Reference  Model. 

1.  Portable  Operating  System  Interface  for  Computer  Environments  (POSIX). 

As  of  mid  September,  1992,  over  70  implementations  have  been  validated 
for  conformance  to  FIPS  151-1.  On  a quarterly  basis,  validated  POSIX 
implementations  are  listed  in  the  Validated  Products  List.  POSIX 
conformance  testing  is  operated  through  accredited  testing  laboratories, 
of  which  there  are  currently  eight.  A NIST  POSIX  Electronic  Mail  File 
Service  is  available  for  on-line  access  to  the  most  recent  information 
associated  with  the  POSIX  conformance  testing  service.  Via  Internet,  a 
system’s  user  only  needs  to  type  "posix(2)nist.gov"  at  the  mail  level  to 
access  the  POSIX  Electronic  Mail  File  Service.  Current  cost  for  the  test 
suite  (through  NTIS)  is  $2,500. 

The  European  Community  under  the  Conformance  Testing  Services  II 
(CTS2)  initiative  also  has  a POSIX  conformance  testing  service.  They 
chose  the  X/Open  test  suite  VSX  for  the  EC  POSIX  conformance  testing 
services.  Activity  has  been  ongoing  to  harmonize  the  U.S.  NIST  and  EC 
conformance  testing  services. 

X-Open  also  offers  their  conformance  testing  stuie  and  certification  mark 
program  to  those  suppliers  desiring  to  perform  their  own  conformance 
testing. 

2.  Programmer's  Hierarchical  Interactive  Graphics  System  (PHIGS). 

A conformance  testing  service  was  started  at  NIST  October  1,  1992.  This 
conformance  testing  service  uses  version  2 of  the  PHIGS  Validation  Tests 
(PVT).  Both  version  2 and  its  predecessor,  version  1,  were  developed  by 
NIST.  Version  1 was  made  available  in  July  1990,  but  was  used  only  for 
component  testing  by  implementors.  The  NIST  conformance  testing 
service  measures  conformance  of  PHIGS  implementations  to  PHIGS  FIPS 
153. 

Figure  12  depicts  the  processes  associated  with  the  PHIGS  conformance 
testing  process.  Semantic  requirements  are  identified  in  the  PHIGS 
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international  standard  and  formally  restated.  They  describe  the  correct 
behavior  expected  of  PHIGS  functions  and  data.  These  semantic 


Figiire  12:  PHIGS  CONFORMANCE  TESTING  PROCESS 

requirements  logically  support  test  cases.  The  PVT  module  contains  the 
semantic  requirements  and  program  documentation  (including  the  operator 
script)  and  code.  The  program  code  includes  the  test  cases  and  may 
generate  pass/fail  messages;  generate  messages  (other  than  pass/fail): 
prompt  the  operator  and  process  responses;  and  pass  input  parameters  to 
and  receive  output  parameters  from  PHIGS  functions. 

The  operator  reads  the  program  documentation  and  responds  to  the 
prompts,  inspects  graphic  output,  and  generates  graphic  input.  Actual 
PHIGS  functions  generate  the  graphic  output  and  accept  graphic  input  and 
inspect  or  change  the  PHIGS  data.  The  actual  PHIGS  data  will  record  some 
graphical  input. 

The  evaluator  examines  messages  and  may  inspect  module  semantic 
requirements  and  programs,  as  well  as  analyze  the  standard.  Ideal  PHIGS 
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functions  and  data  may  or  may  not  correspond  to  the  lUT's  actual  functions 
and  data. 

3.  X-Window. 

The  U.S.  Government  will  provide  third-party  conformance  testing 

services  through  NVLAP  when  test  suites  and  testing  policy  for  FIPS  PUB 
158  are  available.  Such  availability  is  expected  around  1995. 

4.  Integrated  Services  Digital  Network  (ISDN). 

NIST  is  developing  the  abstract  test  suites  for  ISDN  layers  one  through 

three,  which  will  be  included  as  part  of  the  conformance  testing  program 
for  "GOSIP  2"  (FIPS  146-1).  These  ATSs  will  also  cover  the  conformance 
requirements  associated  with  the  proposed  FIPS  for  ISDN.  The 
conformance  testing  service  for  the  ISDN  applications  associated  with 
layers  one  through  three  is  anticipated  by  the  end  of  1992. 

The  European  Community  (EC),  under  the  Conformance  Testing  Service  II 
(CTS2)  activities,  is  also  developing  a conformance  testing  service  for  the 
ISDN  layers.  NIST  and  the  ISDN  CTS2  EC  participants  are  attempting  to 
harmonize  their  abstract  test  suites  through  the  Consultative  Committee 
on  International  Telegraph  and  Telephone  (CCITT).  Currently,  there  is  too 
much  disagreement  to  harmonize  the  layer  one  ATS;  therefore,  the  United 
States  and  Europe  will  each  retain  their  own  for  ISDN  layer  one 
conformance  testing  activities.  Further  agreement  has  been  reached  for 
layers  two  and  three:  Europe  has  adopted  the  U.S.  ATS  for  layer  two,  and 
the  U.S.  has  adopted  the  European  ATS  for  layer  three.  The  CTS2  initiative 
is  working  on  an  ATS  for  layer  four  which  the  United  States  is  considering 
for  adoption  [SU92]. 
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VII.  CONCLUSION. 


Multiple  disparate  activities  have  been  started  and  is  continuing  under  the 
CALS  umbrella  of  "testing."  The  CEIO  continues  to  assess  the  overall 
testing  investment  and  the  continued  requirements  for  the  various  levels 
of  testing  within  the  CALS  community.  Testing  is  a recognized  and 
accepted  process  to  be  performed  at  various  levels  of  standard  and 
product  development  and  enterprise  user  integration: 

{standards}  test  and  develop  the  standard 

{component}  test  and  develop  the  COTS  implementation 

{conformance}  test  and  deploy  the  COTS  implementation 

{acceptance}  test  and  accept  the  COTS  implementation  into  the  user 
enterprise 

These  levels  of  testing  activity  are  dependent  on  one  another.  The 
ultimate  goal  is  to  ensure  the  government  user's  requirements  at  the 
system  procurement  level  have  been  met.  Without  the  cumulative  support 
from  strong  standards  testing,  component  testing,  and  conformance 
testing  programs,  the  effort  and  cost  associated  with  CALS  acceptance 
testing  would  be  redundant  and  inefficient. 

Several  issues  which  face  the  CALS  initiative  were  highlighted  in  this 
report.  The  following  are  a summary  of  the  issues  and  their  associated 
recommendations.  The  CEIO  should: 

Issue.  Whether  the  CEIO  should  recognize  any  acceptance  testing 
performed  by  the  systems  integrator  prior  to  government  procurement. 
Recommendation.  The  government  user  applies  predefined  requirements 
which  the  systems  integrator  applies  in-house,  and  the  government  user 
accepts  the  results. 

Issue.  Whether  it  is  cost-effective  for  the  CEIO  to  fund  a validation 
testing  system  for  STEP,  and  if  so,  where  should  such  a system  be  hosted. 
Recommendation.  Sustain  a Validation  Testing  System  for  STEP  at  NIST. 
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Issue.  Although  NIST  develops  conformance  testing  services  for  some 
standards  which  are  Federal  Information  Processing  Standards 
Publications  (FIPS  PUBS),  no  means  exists  to  perform  conformance  testing 
on  the  additional  requirements  imposed  on  the  supplier  through  military 
specifications.  Should  NIST  be  responsible  for  CALS  military 
specification  conformance  testing? 

Recommendation.  Sponsor  NIST  to  provide  conformance  testing  services 
for  the  CALS  military  specifications;  funding  conformance  testing  service 
development  for  both  the  FIPS  and  associated  military  specification  if 
necessary. 

Issue.  Accreditation  of  testing  laboratories  is  often  viewed  as  an 
expensive  overhead,  both  in  actual  dollars  required  to  initiate,  and  time 
required  to  establish  such  a program.  Should  the  CEIO  invest  in 
accreditation? 

Recommendation.  Support  the  use  of  accreditation  for  first,  second,  and 
third  party  testing  laboratories  through  NVLAP. 

Issue.  The  CALS  Test  Network  (CTN)  affirms  CALS  compliance  only  for 
DoD-owned  systems.  Is  this  appropriate? 

Recommendation.  Remove  any  conformance  testing  from  the  CTN 
responsibilities,  especially  if  it  is  limited  only  to  DoD-owned  systems. 

Issue.  The  CTN  performs  standards  testing  and  interoperability  testing 
without  routinely  performing  conformance  testing.  This  CTN  practice  is 
not  the  same  sequence  that  occurs  in  general  practice. 
Recommendation.  Advocate  the  CTN  for  only  standards  testing; 
interoperability  testing  by  the  CTN  should  be  performed  only  as  part  of 
acceptance  testing  for  a given  procurement. 

Issue.  Should  the  CEIO  try  to  leverage  industry's  test  suite  and  tool 
development  resources  to  benefit  conformance  testing  and  reduce 
industries'  investment  costs? 

Recommendation.  Encourage  supplier  or  enterprise  user-developed 
proprietary  testing  tools  to  be  made  available  to  NIST  for  CALS 
conformance  testing  programs  for:  (1)  use  if  complete,  or  (2)  adaptation 
if  incomplete. 
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Issue.  Even  if  conformance  testing  services  existed  for  all  the  CALS 
military  specifications,  how  does  the  CEIO  guarantee  enterprise  users 
will  take  advantage  of  such  services? 

Recommendation.  Instill  conformance  testing  into  the  infrastructure 
through  changes  to  the  Federal  and  Defense  Acquisition  Regulations; 
develop  policy  and  guidelines  which  implement  conformance  testing  into 
the  purchasing  infrastructure:  provide  funding  to  establish  and  ensure 
conformance  testing  services  exist  for  each  of  the  CALS  specifications: 
and  leverage  existing  commercial  efforts  and  any  commercial  demands  for 
conformance  testing. 

The  world  of  information  technology  standards  is  not  a finite,  bounded 
environment  where  decisions  can  be  made  and  proven  with  mathematical 
precision.  The  CEIO  must  depend  on  consensus  building  to  acquire 
standards  for  the  CALS  community  which  reflect  the  functionality 
necessary  to  meet  DoD  requirements.  Testing  activities  and  supportive 
terminology  and  policies  can  also  be  selected  from  many  correct 
alternatives.  This  report  has  highlighted  some  of  the  most  commonly 
recognized  terminology  and  processes,  and  supports  the  CEIO  in  choosing 
what  best  meets  its  environment.  The  CEIO  must  assess  the  technical, 
political,  and  economic  ramifications  of  its  decisions  for  any  given 
weapon  system  acquisition  or  supporting  life  cycle  program. 
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Appendix  A:  Terminology  and  Acronyms 


A.  Terminology. 

abstract  test  method:  The  description  of  how  an  lUT  is  to  be  tested, 
given  at  an  appropriate  level  of  abstraction  to  make  the  description 
independent  of  any  particular  realization  of  the  means  of  testing,  but  with 
enough  detail  to  enable  tests  to  be  specified  for  this  test  method  [9646- 
1]. 

abstract  test  suite:  A complete  set  of  abstract  test  cases,  possibly 
combined  into  nested  abstract  test  groups,  that  is  necessary  to  perform 
conformance  testing  for  a standard  or  group  of  standards  [31-92]. 

acceptance  test:  (ISO  2382,  Vocabulary  - Information  Processinol  A 
test  of  a system  or  functional  unit,  usually  performed  by  users  on  their 
premises  after  installation,  with  the  participation  of  the  vendor  to  ensure 
that  contractual  requirements  are  met  [FIPS11-3]. 

{user}  acceptance  testing:  Determines  whether  a software  system 
satisfies  its  acceptance  criteria  and  enables  the  user  to  determine 
whether  to  accept  the  system.  This  includes  the  planning  and  execution  of 
several  kinds  of  tests  (e.g.,  functional,  volume,  performance  tests)  to 
demonstrate  the  implemented  software  satisfies  the  user  requirements. 
NOTE  - This  does  not  form  part  of  conformance  testing  [31-92]. 

(laboratory)  accreditation:  The  formalized  initial  and  continuing 
process  of  ensuring  a testing  laboratory  is  competent  to  carry  out 
specific  (types  of)  tests.  NOTE  - The  term  "laboratory  accreditation" 
covers  the  recognition  of  both  the  technical  competence  and  the 
impartiality  of  a testing  laboratory.  Accreditation  is  normally  awarded 
following  successful  laboratory  assessment  and  is  followed  by 
appropriate  surveillance  [31-92]. 

accreditation  body:  A body  that  conducts  and  administers  a laboratory 
accreditation  scheme  and  grants  accreditation  [31-92]. 
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application  protocol  (AP);  A part  of  STEP  that  describes  the  use  of 
integrated  resources  to  satisfy  the  scope  and  information  requirements 
for  a specific  application  context  [1-92]. 

application  protocol  validation:  The  process  of  evaluating  a 
candidate  application  protocol  (AP)  and  its  components  to  determine 
whether  these  satisfy  specified  requirements  [AP92]. 

certificate  of  conformity,  certificate  of  conformance:  A 

document  issued  under  the  rules  of  a certification  system  indicating  that 
adequate  confidence  is  provided  that  an  implementation  under  test  is  in 
conformity  with  a specific  standard  or  technical  specification  as 
determined  through  use  of  a specified  test  method  [31-92]. 

certification:  Procedure  by  which  a third  party  gives  written  assurance 
that  a product,  process  or  service  conforms  to  specified  requirements 
[IS02]. 

certification  body:  An  impartial  body  possessing  the  necessary 
competence  and  reliability  to  operate  a certification  system,  and  in  which 
the  interests  of  all  parties  concerned  with  the  function  of  the  system  are 
represented  [31-92]. 

client  (of  a testing  laboratory):  The  organization  that  submits  a 
system  or  implementation  for  conformance  testing  [9646-1]. 

compliance:  The  act  or  process  of  complying  to  a desire,  demand,  or 
proposal  or  to  coercion  [WEB79]. 

component  testing:  Testing  conducted  to  verify  the  implementation  of 
the  design  for  one  software  element  (for  example,  unit,  module)  or  a 
collection  of  software  elements  [IEEE-86]. 

NOTE  - This  does  not  form  part  of  conformance  testing. 

conformance:  See  conformity. 
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conformance  assessment  process:  The  complete  process  of 
accomplishing  all  conformance  testing  activities  necessary  to  determine 
the  conformance  of  an  implementation  [31-92], 

conformance  test  report:  A document  written  at  the  end  of  the 
conformance  assessment  process,  which  provides  both  summary  and 
detailed  information  [31-92  generalizeds], 

conformance  testing:  Testing  the  extent  to  which  an  lUT  is  a 
conforming  implementation  [9646-1], 

conformity,  conformance:  Fulfillment  by  a product,  process,  or  service 
of  specified  requirements  [IS02], 

enterprise  user.  An  organization  or  person  who  builds,  uses,  maintains, 
or  disposes  of  information  generated  from  an  implementation, 

executable  test  suite:  A complete  set  of  executable  test  cases  (an 
instantiation  of  an  abstract  test  case  with  values)  that  is  necessary  to 
perform  conformance  testing  for  a standard  or  group  of  standards  [31-92], 
first  party  testing  laboratory:  See  third  party  testing  laboratory. 

Implementation  Under  Test:  That  part  of  a product  which  is  to  be 
studied  under  testing,  which  should  be  an  implementation  of  one  or  more 
characteristics  of  the  standard(s)  [31-92  generalized], 

implementor:  See  supplier, 

interoperability  testing:  Related  to  acceptance  testing,  but  applied  to 
the  examination  of  the  information  exchange  and  sharing  between  two 
specific  implementations  under  test  (lUT)  and  the  ability  of  each  lUT  to 
use  such  information,  NOTE  - This  does  not  form  part  of  conformance 
testing  [31-92], 


9 Those  definitions  referenced  in  this  manner  have  been  generalized 
since  the  definition  contained  text  specific  to  STEP, 
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means  of  testing:  The  combination  of  equipment  and  procedures  that  can 
perform  the  derivation,  selection,  parameterization,  and  execution  of  test 
cases,  in  conformance  with  a reference  standardized  abstract  test  suite, 
and  can  produce  a conformance  log  [9646-1]. 

military  specification  testing:  The  processes  associated  with 
validation  (definition  relative  to  standards  testing)  of  the  military 
specifications. 

performance  testing:  Measures  the  performance  characteristics  of  an 
lUT,  such  as  its  throughput,  response  time,  number  of  transactions,  and 
responsiveness  under  various  conditions.  Note  - This  does  not  form  part 
of  conformance  testing  [31-92]. 

proficiency  testing  (laboratory):  Determination  of  laboratory  testing 
performance  by  means  of  inter-laboratory  test  comparisons  [3768-88]. 

protocol  Implementation  conformance  statement  (PICS):  A 

statement  made  by  the  supplier  of  an  implementation  or  system,  stating 
which  capabilities  have  been  implemented  for  a given  standard  [9646-1]. 

protocol  implementation  extra  information  for  testing  (PIXIT): 

A statement  made  by  a supplier  or  implementor  of  an  lUT  which  contains 
or  references  all  of  the  information  (in  addition  to  that  given  in  the  PICS) 
related  to  the  lUT  and  its  testing  environment,  which  will  enable  the 
testing  laboratory  to  run  an  appropriate  test  suite  against  the  lUT  [9646- 
1]- 

robustness  testing:  Determines  how  well  an  lUT  recovers  (for  example: 
from  various  error  conditions)  [31-92]. 

second  party  testing  laboratory:  See  third-party  testing  laboratory. 

self-certification:  term  deprecated.  See  "supplier’s  declaration"  [IS02]. 

standards  testing:  Determines  whether  the  national,  international,  or 
military  standards  (and  specifications)  are  viable  and  implementable. 
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static  conformance  review:  A review  of  the  extent  to  which  the 
static  conformance  requirements  are  met  by  the  lUT,  accomplished  by 
comparing  the  PICS  with  the  static  conformance  requirements  expressed 
in  the  relevant  international  standard(s)  or  CCITT  recommendation(s) 
[9646-1]. 

supplier,  vendor,  Implementor:  An  organization  or  individual  who 
develops  commercial  off-the-shelf  implementations. 

supplier's  declaration:  Procedure  by  which  a supplier  gives  written 
assurance  that  a product,  process  or  service  conforms  to  specified 
requirements.  Note  - In  order  to  avoid  confusion,  the  expression  "self- 
certification"  should  not  be  used  [IS02]. 

System  Under  Test:  The  computer  hardware,  software,  and 

communication  network  required  to  support  the  lUT  [31-92]. 

test  campaign:  The  process  of  running  the  executable  test  suite  for  a 
particular  lUT  [9646-1]. 

test  method:  Specified  technical  procedure  for  performing  a test  [IS02]. 
Specified  technical  procedure  for  performing  a testing  service,  including 
the  specification  of  the  individual  test  cases  which  comprise  a test  suite; 
the  test  tools  (both  hardware  and  software)  used  to  run  those  executable 
test  cases  and  the  way  in  which  those  test  tools  are  used;  and  the 
procedures  used  to  select  and  run  the  test  cases  and  to  analyze  the 
observations  and  state  the  test  results  [EW15]. 

test  tool:  Hardware  and/or  software,  excluding  the  test  suite  itself, 
which  is  run  under  the  control  of  the  testing  laboratory,  in  order  to  carry 
out,  or  assist  in  carrying  out,  the  testing  required  [EW15]. 

testing:  Action  of  carrying  out  one  or  more  tests  (Technical  operation 
that  consists  of  the  determination  of  one  or  more  characteristics  of  a 
given  product,  process  or  service  according  to  a specified  procedure.) 
[IS02]. 
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testing  laboratory:  An  organization  that  carries  out  the  conformance 
assessment  process.  This  can  be  a third  party,  a user  organization,  or  an 
identifiable  part  of  the  supplier  organization  [31-92]. 

third  party  certification  program:  An  organized  system  (1)  under 
which  similar  products  or  services  of  any  number  of  producers  may  be 
certified  as  conforming  to  the  referenced  standards  or  specifications  on  a 
uniform  and  equitable  basis,  (2)  which  uses  or  is  operated  by  a third-party 
testing  laboratory,  and  (3)  which  authorizes  the  use  of  controlled 
certification  marks  or  certificates  of  conformity  as  evidence  of 
conformity  [Z34.1]. 

third  party  testing  laboratory:  Person  or  body  that  is  recognized  as 
being  independent  of  the  parties  involved,  as  concerns  the  issue  in 
question.  NOTE  - Parties  involved  are  usually  supplier  ("first  party")  and 
purchaser  ("second  party")  interests  [IS02]. 

validation  (definition  relative  to  standards  testing):  An  act, 
process,  or  instance  of  making  valid.  (Valid  - having  legal  efficacy  or 
force.)  [WEB79] 

NOTE  - This  does  not  form  part  of  conformance  testing. 

validation  (definition  relative  to  conformance  testing):  The 

process  of  checking  the  conformity  of  an  implementation  of  a standard  to 
its  standard  specification  through  conformance  testing,  and  when 
compliance  is  demonstrated,  issuing  a validation  certificate  [FIPS88]. 

vendor:  See  supplier. 


B.  Acronyms. 

ANS:  American  National  Standard 

ANSI:  American  National  Standards  Institute 

CAD:  Computer-Aided  Design 

CALS:  Computer-aided  Acquisition  and  Logistic  Support 
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CCITT: 

CD; 

CE: 

CEC: 

CEIO: 

CITIS; 

CGM; 

CIM: 

COTS: 

CTN: 

CTS: 

DLA: 

DSREDS 

EC: 

EDCARS: 

EDIF: 

EDMICS: 

GCA: 

GKS: 

GOSIP: 

JITC: 

lEC; 

IGES: 

IPC: 

IPO: 

IRDS: 

ISG: 

ISO: 

ITI; 

lUT: 

MOU: 

NIST: 

NPT: 

NSIA: 

PDES: 

PHIGS; 

PICS: 


Consultative  Committee  on  International  Telegraph  and 

Telephone 

Committee  Draft 

Concurrent  Engineering 

Commission  for  European  Community 

CALS  Evaluation  and  Integration  Office 

Contractor's  Integrated  Technical  Information  Service 

Computer  Graphics  Metafile 

Corporate  Information  Management 

Commercial  Off-The-Shelf 

CALS  Test  Network 

Conformance  Testing  Service 

Defense  Logistics  Agency 

Digital  Storage  and  Retrieval  Engineering  Data  System 
European  Community 

Engineering  Data  Computer  Assisted  Retrieval  System 
Electronic  Design  Interchange  Format 

Engineering  Data  Management  Information  and  Control  System 
Graphic  Communications  Association 
Graphical  Kernel  System 

Government  Open  Systems  Interconnection  Profile 
Joint  Interoperability  Test  Center 
International  Electrotechnical  Commission 
Initial  Graphics  Exchange  Specification 
Institute  for  Packaging  of  Electronic  Components 
IGES/PDES  Organization 
Information  Resource  Dictionary  System 
Industry  Steering  Group 

International  Organization  for  Standardization 

Industrial  Technical  Institute 

Implementation  Under  Test 

Memorandum  of  Understanding 

National  Institute  of  Standards  and  Technology 

National  PDES  Testbed 

National  Security  Industrial  Association 

Product  Data  Exchange  using  STEP 

Programmer's  Hierarchical  Interactive  Graphics  System 

Protocol  Implementation  Conformance  Statement 
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PIXIT; 

Protocol  Implementation  Extra  Information  for  Testing 

POSIX: 

Portable  Operating  System  Interface  for 
Environments 

Computer 

R&D: 

Research  and  Development 

RFP: 

Request  for  Proposal 

SGML; 

Standard  Generalized  Markup  Language 

STEP: 

Standard  for  the  Exchange  of  Product  Model  Data 

SUT: 

System  Under  Test 

VHDL: 

Very  High  Speed  Integrated  Circuit  (VHSIC) 
Definition  Language 

Hardware 
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Appendix  B:  Existing  Directives  and  Guidelines 


Standards  Testing. 

CTN  91-042,  CALS  Test  Network  Handbook. 

This  series  of  NIST  publications  are  specific  to  validating  various  aspects 
of  STEP: 

NISTIR  4417,  Development  Plan:  Validation  Testing  System. 

NISTIR  4636,  Validation  Testing  System  Requirements. 

NISTIR  4684,  A Proposed  Testing  Methodology  for  STEP  Application 
Protocol  Validation. 

NISTIR  4735,  Validating  STEP  Application  Models  at  the  National 
PDES  Testbed. 

NISTIR  4742,  Architecture  for  the  Validation  Testing  System 
Software. 

Component  Testing. 

ANSI/IEEE  Std  1012-1986,  IEEE  Standard  for  Software  Verification 
and  Validation  Plans. 

DOD-STD-2167A,  Defense  System  Software  Development. 

MIL-HDBK-287,  Tailoring  Guide  for  DOD-STD-2167A.  Defense  System 
Software  Development. 

DOD-STD-2168,  Defense  System  Software  Quality  Program. 

MIL-HDBK-2168,  A Guide  for  DOD-STD-2168.  Defense  System 
Software  Development. 
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FIPS  PUB  101,  Guideline  for  Lifecycle  Validation.  Verification,  and 
Testing  of  Computer  Software. 

FIPS  PUB  132,  Guideline  for  Software  Verification  and  Validation 

Elan?.. 

NIST  Special  Publication  500-165,  Software  Verification  and 
Validation:  Its  Role  in  Computer  Assurance  and  Its  Relationship  with 

Software  Project  Management  Standards. 

MIL-HDBK-59,  Department  of  Defense  Computer-aided  Acouisition 
and  Logistic  Support  fCALSI  Program  Implementation  Guide. 

R.  Ebert,  J.  Lugger,  and  R.  Goecke,  Practice  in  Software  Adaption  and 
Maintenance.  North-Holland  Publishing  Company,  New  York,  1980. 

Abbott,  J.,  Software  Validation.  A Study  of  Techniques  and  Methods. 
National  Computing  Centre  Limited,  1983. 

Gerrard,  Christopher  Paul,  Effective  Software  Testing.  The  National 
Computing  Centre  Limited,  1987. 

Hetzel,  William,  The  Complete  Guide  to  Software  Testing.  QED 
Information  Sciences,  Inc.,  Wessley  MA,  1984. 

Richard  DeMillo,  W.  Michael  McCracken,  R.J.  Martin,  and  John 
Passafiume,  Software  Testing  and  Evaluation.  The 
Benjamin/Cummings  Publishing  Company,  Inc.,  Reading  MA,  1987. 

Conformance  Testing. 

ISO/IEC  Guide  2,  General  Terms  and  Their  Definitions  Concerning 
Standardization  and  Related  Activities. 

ISO/IEC  Guide  7,  Requirements  for  Standards  Suitable  for  Product 
Certification. 
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ISO/IEC  Guide  16,  Code  of  Principles  on  Third-Partv  Certification 
Systems  and  Related  Standards. 


ISO/IEC  Guide  23,  Methods  of  Indicating  Conformity  with  Standards 
for  Third-Party  Certification  Systems. 

ISO/IEC  Guide  25,  General  Requirements  for  the  Technical 

Competence  of  Testing  Laboratories. 

EW  15  and  ECITC  N239,  Draft  Interpretation  of  Accreditation 
Requirements  as  Specified  in  ISO/IEC  Guide  25  and  EN  45001  for 

Information  Technology  Test  Laboratories  for  Software  and 

Communications  Testing  Seryices.  Version  1.1,  20  Noyember  1991. 

ISO/IEC  Guide  27,  Guidelines  for  Corrective  Action  to  be  Taken  by  a 
Certification  Body  in  the  Event  of  Either  Misapplication  of  its  Mark 

of  Conformity  to  a Product,  or  Products  which  bear  the  Mark  of  the 

Certification  Body  being  Found  to  Subject  Persons  or  Property  to 

Risk. 

ISO/IEC  Guide  38,  General  Requirements  for  the  Acceptance  of 

Testing  Laboratories. 

ISO/IEC  Guide  40,  General  Requirements  for  the  Acceptance  of 

Certification  Bodies. 

ISO/IEC  Guide  45,  Guidelines  for  the  Presentation  of  Test  Results. 

European  Norm,  EN45000  series:  These  "European  Norm"  standards 
relate  to  criteria  for  accrediting  testing  laboratories,  operating 
laboratory  accreditation  bodies  and  certification  bodies,  and 
performing  supplier’s  declaration  of  conformity. 

IS  9646-1  through  5:  This  series  of  standards  pertain  to 

conformance  testing  requirements  specific  to  GOSIP. 

ISO  "CD"  10303-31  through  34:  This  series  of  working  draft  and 
committee  draft  standards  pertain  to  conformance  testing 
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requirements  specific  to  STEP.  They  are  adapted  from  the  IS  9646 
GOSIP  series. 

ISO/IEC  DIS  10641,  Conformance  Testing  of  Implementations  of 
Graphics  Standards.  This  standard  pertains  to  conformance  testing 
requirements  and  procedures  specific  to  the  graphics  standards,  e.g., 
CGM,  PHIGS,  GKS. 

Proposed  American  National  Standard,  Information  Technology  - 
and  Office  Systems  - Conformance  Testing  for  Standard  Generalized 

Markup  Language  fSGMU  Systems.  July  1991.  (ISO/IEC  JTC1/SC18 
approved  NWI  to  develop  same  as  IS) 

ANSI  Z34.1-87,  Third-Partv  Certification  Program.  (Adopted  for  DoD 
use  March  1,  1988.) 

ANSI  Z34.2-87,  Self-Certification  bv  Producer  or  Supplier.  (Adopted 
for  DoD  use  March  1,  1988.) 

NISTIR  4739,  Validated  Products  List.  1992  No.1. 

NISTIR  4576,  Laboratory  Accreditation  in  the  United  States. 

CALS/CE  ISG  Report  #1,  "CALS  ISG  Ad  Hoc  Working  Group  on  CALS 
Testing." 

CALS/CE  ISG  Report  #2,  "Current  Status  and  Responsibilities." 

CALS/CE  ISG  Final  Report,  "CALS/CE  ISG  AD  Hoc  Committee  on 
Testing." 

Acceptance  Testing. 

ISO/IEC  Guide  36,  Preparation  of  Standard  Methods  of  Measuring 
Performance  ('SMMP')  of  Consumer  Goods. 

ISO/IEC  Guide  46,  Comparative  Testing  of  Consumer  Products  & 
Related  Services  - General  Principles. 
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FIPS  75,  Guideline  on  Constructing  Benchmarks  for  ADP  System 
Acquisitions.  September  18,  1980. 

CTN  Report  91-021,  "Digital  Data  Acceptance/Quaiity  Assurance 
Procedures,”  31  March  1992. 

CTN  Report  91-023,  "Field  Testing  of  Phase  I Data  Acceptance 
Procedures." 

CTN  Test  Report  91-027,  "Computer  Assisted  Data  Acceptance 
Procedures,"  31  March  1992. 

CTN  Test  Report  91-028,  "Model  --  Technical  Manual  Data,"  31  March 
1992. 

CALS/CE  ISG  Report,  "Delivery  Verification  and  Acceptance  Testing 
Guideline." 

USAMC  Materiel  Readiness  Support  Activity,  "Validation  Guide,  LSAR 
ADP  Systems,"  January  1987. 

NBS  Special  Publication  500-136,  An  Overview  of  Computer 
Software  Acceptance  Testing. 

NIST  Special  Publication  500-180,  Guide  to  Software  Acceptance. 
IPO,  "Interoperability  Testing  Methodology  IGES  Guidelines." 
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INDEX 


abstract  test  suite  61,  65,  73,  87,  90 

acceptance  testing  ...  iii-iv,  1,  3,  7,  9,  12,  15-16,  27-32,  35-36,  39-40, 

42-46,  75-76,  87,  89,  99 

accreditation  ...  iv,  24-25,  36-40,  47,  51,  55,  57,  76,  79-80,  87,  97-98 

accreditation  body  55,  87 

American  National  Standards  Institute  3,  9-10,  92 

ANSI  3-4,  9-11,  64,  81,  83,  92,  95,  98 

application  protocol  validation  88,  95 

ATS  65,  73 

CAD  22,  55,  63,  66,  92 

CALS  ...  1,  iii-iv,  1-2,  4,  7-14,  16-17,  19-23,  26-29,  32-36,  38-39,  41, 

43-53,  55-56,  60,  65,  71,  75-77,  80-82,  92-93,  95-96, 

98-99 


CALS  Evaluation  and  Integration  Office  iii,  1,  9,  55,  93 

CALS  Test  Network  iv,  4,  8,  19,  22-23,  27,  76,  80,  82,  93,  95 

CCITT  73,  91,  93 

CEIO  ...  iii-iv,  1,  9,  12,  20,  25-26,  29,  31-45,  47-53,  55,  57,  63,  75-77, 

93 

certificate  of  conformance  88 

certificate  of  conformity  65 

certification  ....  iv,  3,  24,  26-27,  38,  64,  71,  81,  83,  88,  90-92,  96-98 


certification  body  

COM  

CIM  

CITIS  

component  testing  

Computer  Graphics  Metafile  . . . . 
conformance  assessment  process 

conformance  test  report  

conformance  testing  . . . iii-iv, 

34-36,  38-53, 

conformance  testing  service 
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10-11,  23,  38,  59-60,  85,  93,  98 

29,  35,  71,  93 

27,  61,  93 

iii,  4-5,  12,  14,  32,  39,  45,  47,  71,  75,  88 

10-11,  59,  93 

61,  89,  92 

15,  89 

2-7,  9,  12,  15-16,  20,  22-23,  25-26,  32, 
55-57,  59-61,  63-71,  73,  75-77,  79-82, 

87-90.  92-93,  96-98 
. V,  2,  23,  35,  40.  44-49.  55-57,  59,  61, 
63,  65-68,  70-71,  73,  76,  79,  93 


Consultative  Committee  on  International  Telegraph  and  Telephone  ...  73, 

93 
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93 

Corporate  Information  Management  29,  93 


COTS  36,  47,  52,  75,  93 

CTN  iv,  4,  19-20,  22,  26-27,  31,  40-45,  76,  82,  93,  95,  99 

CTS  66,  93 

DoD  iv,  1,  3-5,  11-12,  19-20,  23,  29,  31-36,  42-45,  47-48,  50,  52, 

76-77,  79-80,  95,  98 

EDIF  63,  93 


executable  test  suite 


56,  62-63,  6S, 


o-< 


first  party  testing  laboratory  89 

GKS  38,  93,  98 

GOSIP  6,  9-10,  25,  61,  70,  73,  93,  97-98 

Government  Open  Systems  Interconnection  Profile  6,61,93 

Graphical  Kernel  System  38,  93 

lEC  3,  10,  64-65,  80-81,  93,  96-98 

IGES  11,  22-23,  53,  55,  81,  93,  99 


Implementation  Under  Test  iii,  57,  88-89,  93 

Industry  Steering  Group  1,  7,  19-20,  26-27,  93 

Information  Resource  Dictionary  System  68,  93 

Initial  Graphics  Exchange  Specification  11,  23,  55,  93 

International  Electrotechnical  Commission  3,  10,  93 

International  Organization  for  Standardization  3,  9,  21,  93 

interoperability  testing  iv,  9,  16,  27,  36,  40-45,  76,  81,  99 

IPC  63-65,  81,  93 

IPO  22,  93,  99 

IRDS  68-70,  80,  85,  93 

ISG 1,  19-20,  27,  81,  83,  93,  98-99 

ISO 3,  9-11,  21-22,  26,  56,  59,  79-81,  87,  93,  96-98 

lUT  57,  59,  73,  87,  89-91,  93 

JITC  35,  61,  93 

military  specification  testing  12-14, 90 

National  Institute  of  Standards  and  Technology  ...  iii,  10,  19-21,  82,  93 

National  PDES  Testbed  4,  8,  21,  25,  33,  65-66,  79,  93,  95 

NIST  ..  iii-iv,  4,  9-10,  19-21,  23-24,  26,  32-36,  43-44,  47,  51,  55-59, 

65,  67-71,  73,  75-76,  79-80,  82-83,  93,  95-96,  99 

NPT  4,  21-22,  25,  93 

NTIS  39,  71 

NVLAP  24-25,  36-37,  40,  61,  73,  76,  79 
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PDES  4,  8,  21-22,  25,  33,  55,  65-66,  79,  81,  93,  95 


PHIGS  71-72,  82,  85,  93,  98 

PICS  6,61-62,90-91,93 

PIXIT  61-62,  90,  94 

Portable  Operating  System  Interface  for  Computer  Environments  ....  71, 

94 

POSIX  25,  71,  85,  94 

Product  Data  Exchange  using  STEP  93 

second  party  testing  laboratory  90 

SGML  10-11,  51,  53,  56,  94,  98 

Standard  for  the  Exchange  of  Product  Model  Data  iv,  6,  65,  94 

Standard  Generalized  Markup  Language  10-11,  56,  94,  98 

standards  testing  ....  iii-iv,  4,  12-14,  20-22,  25,  34-35,  40-45,  75-76, 

90,  92,  95 

STEP  iv,  6-10,  13,  21-22,  25-26,  31-34,  55,  65-68,  75,  79-80,  85, 

88,  93-95,  98 

SUT  61-62,94 

System  Under  Test  94 

technical  information  11,  19,  27,  39,  55,  61,  79,  93 

testing  laboratory  ..  4,  15,  24-25,  35-38,  40,  51,  57,  59-60,  64,  87-92 

third  party  certification  program  83,  92,  98 

third  party  testing  laboratory  4,  89-90,  92 

Validated  Products  List  51,  98 

validation  iv,  4-5,  21-22,  26,  31-32,  34,  56-57,  71,  75,  79,  81,  88, 

90,  92,  95-96,  99 

verification  iv,  81,  95-96,  99 

VHDL  63,  94 
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